Home / Flink / Checkpoints and state
Flink: Exceeded checkpoint tolerable failure threshold
org.apache.flink.util.FlinkRuntimeExceptionExceeded checkpoint tolerable failure threshold.The job failed because more checkpoints failed in a row than execution.checkpointing.tolerable-failed-checkpoints allows. The default is 0, so the first failed checkpoint restarts the job. The real cause is the checkpoint failure logged before this message.
Common causes
- Checkpoints are expiring because of backpressure or slow state uploads.How to confirm: The JobManager log shows "Checkpoint expired before completing." for the same checkpoint IDs just before this error.
- A task failed to snapshot its state, for example a serialization error or a sink that could not pre-commit.How to confirm: Search the TaskManager logs for "Asynchronous task checkpoint failed" or a stack trace with the failing checkpoint ID.
- Checkpoint storage is unavailable or full (HDFS quota, S3 permissions).How to confirm: The logs show IOException or access errors from the checkpoint storage path.
How to fix it
Find and fix the underlying checkpoint failure first; this message only reports that the limit was reached.
Allow a few failed checkpoints so one slow checkpoint does not restart the job. The job keeps running and recovers from the last successful checkpoint if it later fails.
| Setting | Default | Note |
|---|---|---|
execution.checkpointing.tolerable-failed-checkpoints | 0 |
Where the message comes from
CheckpointFailureManager.javainapache/flinkatrelease-2.3.0- Message checked in Flink 2.3.0