Home / Flink / Resources and memory
Flink: IOException: Insufficient number of network buffers
java.io.IOExceptionInsufficient number of network buffers: required 2049, but only 1023 available. The total number of network buffers is currently set to 2048 of 32768 bytes each.A task could not reserve the network buffers it needs for its input and output channels. The network memory pool of the TaskManager is too small for the parallelism and the number of shuffles running on it.
Common causes
- High parallelism with all-to-all shuffles (keyBy, rebalance): the number of channels grows with the square of the parallelism.How to confirm: The required number in the message grows when you raise parallelism.
- Many slots per TaskManager share one network memory pool.How to confirm: taskmanager.numberOfTaskSlots is high while network memory is at the default fraction.
How to fix it
Give the TaskManager more network memory.
| Setting | Default | Note |
|---|---|---|
taskmanager.memory.network.fraction | 0.1 | |
taskmanager.memory.network.min | 64m | |
taskmanager.memory.network.max | Long.MAX_VALUE bytes |
Or reduce demand: lower parallelism for the shuffle-heavy part of the job, or use fewer slots per TaskManager.
Where the message comes from
NetworkBufferPool.javainapache/flinkatrelease-2.3.0- Message checked in Flink 2.3.0
Message template in the source: Insufficient number of network buffers: required %d, but only %d available. %s.