Checkpoints?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Checkpoints?

Steve Jerman
Hi,

Configuration:
Flink 1.2.0
I'm using the Rocks DB backend for checkpointing.

The problem I have is that no checkpoints are being deleted, and my disk is filling up.

Is there configuration for this?

Thanks
Steve
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Checkpoints?

Till Rohrmann
Hi Steve,

in the past we had some problems with cleaning up old checkpoints. But this was in 1.1.x. These problems should be fixed by now.

Could you try upgrading to Flink 1.2.1 in order to see whether the problem persists? If this is the case, then it would be great if you could share the JobManager logs on debug log level with us.

How long is your checkpoint interval? Deleting files from HDFS/S3 can take some time and if the checkpoint interval is shorter than this time, then the system won't be able to delete old checkpoints quick enough.

Cheers,
Till

On Fri, Jun 2, 2017 at 5:57 PM, Steve Jerman <[hidden email]> wrote:
Hi,

Configuration:
Flink 1.2.0
I'm using the Rocks DB backend for checkpointing.

The problem I have is that no checkpoints are being deleted, and my disk is filling up.

Is there configuration for this?

Thanks
Steve

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Checkpoints?

Steve Jerman
Thanks. Upgraded to 1.2.1 - problem goes away 

Steve

On Jun 2, 2017, at 10:08 AM, Till Rohrmann <[hidden email]> wrote:

Hi Steve,

in the past we had some problems with cleaning up old checkpoints. But this was in 1.1.x. These problems should be fixed by now.

Could you try upgrading to Flink 1.2.1 in order to see whether the problem persists? If this is the case, then it would be great if you could share the JobManager logs on debug log level with us.

How long is your checkpoint interval? Deleting files from HDFS/S3 can take some time and if the checkpoint interval is shorter than this time, then the system won't be able to delete old checkpoints quick enough.

Cheers,
Till

On Fri, Jun 2, 2017 at 5:57 PM, Steve Jerman <[hidden email]> wrote:
Hi,

Configuration:
Flink 1.2.0
I'm using the Rocks DB backend for checkpointing.

The problem I have is that no checkpoints are being deleted, and my disk is filling up.

Is there configuration for this?

Thanks
Steve


Loading...