Kafka consumer: CommitFailedException: Commit cannot be completed since the group has already rebalanced
org.apache.kafka.clients.consumer.CommitFailedExceptionCommit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.The consumer took longer than max.poll.interval.ms (5 minutes by default) between two poll() calls, so the group coordinator considered it dead and gave its partitions to another member. Its offset commit is rejected, and the records it processed will be consumed again.
Common causes
- Processing one batch from poll() takes too long, for example slow database writes or external calls per record.How to confirm: Log the time between poll() calls; it exceeds max.poll.interval.ms right before the error.
- A long garbage collection pause or a blocked thread stopped the poll loop.How to confirm: GC logs or a thread dump show a pause or a blocked consumer thread.
How to fix it
Process fewer records per poll so each loop finishes in time.
| Setting | Default | Note |
|---|---|---|
max.poll.records | 500 |
Allow more time between polls when processing is legitimately slow.
| Setting | Default | Note |
|---|---|---|
max.poll.interval.ms | 300000 |
Move slow work to a separate thread pool and keep calling poll(), pausing partitions with pause() and resume() if needed.
Where the message comes from
CommitFailedException.javainapache/kafkaat4.3.2- Message checked in Kafka clients 4.3.2