Home / Spark

Apache Spark errors and exceptions

Common Apache Spark errors and exceptions with their exact messages, causes, ways to confirm them and fixes.

Resources and memory

Spark: Container killed by YARN for exceeding physical memory limitsYARN killed an executor because the whole process (JVM heap plus off-heap memory) used more than the container size. The heap may be fine; the off-heap part,…
Spark: Total size of serialized results is bigger than spark.driver.maxResultSizeThe tasks of one job sent more result data back to the driver than spark.driver.maxResultSize (1g by default) allows, so the job was aborted. collect(),…

Spark: org.apache.spark.SparkException: Task not serializableA function passed to a Spark transformation captures an object that cannot be serialized, so Spark cannot ship the closure to executors. The "Caused by:…

Spark: Could not execute broadcast in 300 secsA broadcast join could not build and send its small side within spark.sql.broadcastTimeout (300 seconds by default). Either the "small" table is not small,…
Spark: MetadataFetchFailedException: Missing an output location for shuffleA reducer asked for map output that no longer has a known location, usually because the executor that wrote it was lost. Spark retries the stage, but…