Home / Flink / Network and heartbeats
Flink: The heartbeat of TaskManager with id ... timed out
java.util.concurrent.TimeoutExceptionThe heartbeat of TaskManager with id container_1700000000000_0001_01_000002(host:8026) timed out.The ResourceManager did not hear from a TaskManager within heartbeat.timeout (50 seconds by default) and released it, failing the tasks on it. The TaskManager usually died, froze in a long garbage collection, or lost network connectivity.
Common causes
- The TaskManager process was killed, for example by the YARN or Kubernetes memory limit (container killed, OOMKilled).How to confirm: Check the container exit code or pod status; exit code 137 or OOMKilled means the process exceeded its memory limit.
- Long garbage collection pauses blocked the heartbeat thread.How to confirm: Enable GC logging on the TaskManager and look for pauses close to the timeout.
- Network problems or an overloaded host.How to confirm: The TaskManager log continues normally but shows connection errors to the JobManager at the same time.
How to fix it
If the container was killed for memory, give the TaskManager more process memory or more managed or overhead memory, depending on which pool is exhausted.
| Setting | Default | Note |
|---|---|---|
taskmanager.memory.process.size | none | |
taskmanager.memory.jvm-overhead.fraction | 0.1 |
Tolerate longer pauses while you investigate GC or network issues.
| Setting | Default | Note |
|---|---|---|
heartbeat.timeout | 50 s | |
heartbeat.interval | 10 s |
Where the message comes from
ResourceManager.javainapache/flinkatrelease-2.3.0- Message checked in Flink 2.3.0
Message template in the source: The heartbeat of TaskManager with id %s timed out.