Kafka producer: RecordTooLargeException: The message is N bytes when serialized which is larger than max.request.size
org.apache.kafka.common.errors.RecordTooLargeExceptionThe message is 1572864 bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration.The producer refused to send a record that is larger than its own max.request.size (1 MiB by default). Raising only the producer limit is not enough: the broker and topic limits must allow the size too.
Common causes
- Individual records are larger than 1 MiB, for example whole documents or images.How to confirm: The message states the serialized size; compare it with the payloads your application sends.
- A serializer bug or unbounded collection makes records unexpectedly large.How to confirm: Log the serialized size of a few records; a few outliers point to bad data rather than a limit that is too low.
How to fix it
Keep large payloads out of Kafka: store them in object storage and send a reference.
If large records are expected, raise every limit on the path: producer, broker or topic, and consumer.
| Setting | Default | Note |
|---|---|---|
max.request.size | 1048576 | producer |
message.max.bytes | 1048588 | broker; max.message.bytes per topic |
max.partition.fetch.bytes | 1048576 | consumer |
Enable compression, which can bring records under the limit.
| Setting | Default | Note |
|---|---|---|
compression.type | none |
Where the message comes from
KafkaProducer.javainapache/kafkaat4.3.2- Message checked in Kafka clients 4.3.2
Message template in the source: The message is %d bytes when serialized which is larger than %d, which is the value of the max.request.size configuration.