Flink, Spark, Kafka and Hive errors explained

Common Flink, Spark, Kafka and Hive errors explained: the exact message, where it is thrown in the source code, likely causes, how to confirm them and how to fix them.

Apache Flink

• Flink: IllegalStateException: Buffer pool is destroyedA task tried to request a network buffer after its buffer pool was released, which happens while the task is being cancelled. This exception is almost always…
• Flink: Checkpoint expired before completingA checkpoint did not finish within execution.checkpointing.timeout (10 minutes by default), so the JobManager aborted it. One expired checkpoint is harmless;…
• Flink: NoResourceAvailableException: Could not acquire the minimum required resourcesThe scheduler waited for task slots and gave up. The job needs more slots than the cluster can provide within slot.request.timeout (5 minutes by default),…
• Flink SQL: Could not find any factory for identifier 'kafka'The 'connector' or 'format' option in a CREATE TABLE statement names a factory that is not on the classpath. Flink discovers connectors with Java SPI, so the…
• Flink: Exceeded checkpoint tolerable failure thresholdThe job failed because more checkpoints failed in a row than execution.checkpointing.tolerable-failed-checkpoints allows. The default is 0, so the first…
• Flink: The heartbeat of TaskManager with id ... timed outThe ResourceManager did not hear from a TaskManager within heartbeat.timeout (50 seconds by default) and released it, failing the tasks on it. The…

Apache Spark

• Spark: Container killed by YARN for exceeding physical memory limitsYARN killed an executor because the whole process (JVM heap plus off-heap memory) used more than the container size. The heap may be fine; the off-heap part,…
• Spark: Could not execute broadcast in 300 secsA broadcast join could not build and send its small side within spark.sql.broadcastTimeout (300 seconds by default). Either the "small" table is not small,…
• Spark: MetadataFetchFailedException: Missing an output location for shuffleA reducer asked for map output that no longer has a known location, usually because the executor that wrote it was lost. Spark retries the stage, but…
• Spark: org.apache.spark.SparkException: Task not serializableA function passed to a Spark transformation captures an object that cannot be serialized, so Spark cannot ship the closure to executors. The "Caused by:…
• Spark: Total size of serialized results is bigger than spark.driver.maxResultSizeThe tasks of one job sent more result data back to the driver than spark.driver.maxResultSize (1g by default) allows, so the job was aborted. collect(),…

Apache Kafka

• Kafka consumer: CommitFailedException: Commit cannot be completed since the group has already rebalancedThe consumer took longer than max.poll.interval.ms (5 minutes by default) between two poll() calls, so the group coordinator considered it dead and gave its…
• Kafka producer: TimeoutException: Expiring N record(s) for topic-partitionRecords waited in the producer longer than delivery.timeout.ms (120 seconds by default) without being acknowledged, so the producer failed them. The producer…
• Kafka: NotLeaderOrFollowerException: the broker is not the current leaderA client sent a request to a broker that is no longer the leader of the partition, usually during a leader election or broker restart. Clients refresh…
• Kafka producer: RecordTooLargeException: The message is N bytes when serialized which is larger than max.request.sizeThe producer refused to send a record that is larger than its own max.request.size (1 MiB by default). Raising only the producer limit is not enough: the…

Apache Hive

• Hive: ParseException: cannot recognize input nearThe parser hit tokens it did not expect. The three tokens quoted after "near" show where parsing stopped; the mistake is at or just before them. Reserved…
• Hive: FAILED: Execution Error, return code 2 from TezTask or MapRedTaskThe query compiled but the job it launched failed. Return code 2 means the Tez DAG or MapReduce job failed at run time. The message itself carries no cause:…
• Hive: SemanticException [Error 10004]: Invalid table alias or column referenceHive could not resolve a column or alias in the query. The message names the reference and lists the columns that were available, which is usually enough to…
• Hive: Number of dynamic partitions created is N, which is more than 1000A dynamic-partition INSERT would create more partitions than hive.exec.max.dynamic.partitions (1000 by default). Before raising the limit, check whether the…
• Hive: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClientHive (or Spark with Hive support) could not create a metastore client. The class name is only the wrapper: the "Caused by" lines underneath say whether the…