dataengref errors

Home / Kafka / Consumer

Kafka consumer: RecordDeserializationException: Error deserializing VALUE for partition

org.apache.kafka.common.errors.RecordDeserializationExceptionError deserializing VALUE for partition orders-2 at offset 18342. If needed, please seek past the record to continue consumption.

The consumer's key or value deserializer failed on one record, so poll() throws and the consumer stops at that offset. It keeps failing on the same record until the code skips it.

Causes · Fixes · Source

Common causes

  1. A producer wrote data in another format (plain JSON into an Avro topic, or a different schema).How to confirm: Read the record at the offset with kafka-console-consumer and look at the raw bytes.
  2. The consumer uses the wrong deserializer for the key or value.How to confirm: The message says KEY or VALUE; compare that side's deserializer with what producers use.

How to fix it

Catch RecordDeserializationException, log or dead-letter the record, and seek past it.

try {
  records = consumer.poll(timeout)
} catch {
  case e: RecordDeserializationException =>
    consumer.seek(e.topicPartition(), e.offset() + 1)
}

Fix the producer or the deserializer so new records match.

Where the message comes from

Message template in the source: Error deserializing %s for partition %s at offset %d. If needed, please seek past the record to continue consumption.

References

Last checked: