Home / Spark / Resources and memory
Spark: Container killed by YARN for exceeding physical memory limits
org.apache.spark.SparkExceptionContainer killed by YARN for exceeding physical memory limits. 9.1 GB of 9 GB physical memory used. Consider boosting spark.executor.memoryOverhead.YARN killed an executor because the whole process (JVM heap plus off-heap memory) used more than the container size. The heap may be fine; the off-heap part, sized by spark.executor.memoryOverhead, is what ran out.
Common causes
- Off-heap usage is larger than the overhead allowance: netty buffers during shuffle, PySpark or pandas UDF workers, native libraries.How to confirm: The executor log has no OutOfMemoryError, but YARN reports the container used more than its limit (for example "9.1 GB of 9 GB physical memory used").
- Very large partitions or skewed keys make single tasks hold much more memory than average.How to confirm: In the Spark UI stage view, a few tasks have far larger "Shuffle Read Size" or "Input Size" than the median.
How to fix it
Increase the overhead directly, or raise the factor so it scales with executor memory.
| Setting | Default | Note |
|---|---|---|
spark.executor.memoryOverhead | executorMemory × 0.10, at least 384 MiB | |
spark.executor.memoryOverheadFactor | 0.10 |
Reduce per-task memory: more shuffle partitions, fewer cores per executor, or fix skew.
| Setting | Default | Note |
|---|---|---|
spark.sql.shuffle.partitions | 200 | |
spark.executor.cores | 1 on YARN |
Where the message comes from
YarnAllocator.scalainapache/sparkatv4.1.3- Message checked in Spark 4.1.3. Before Spark 2.3 the setting was called spark.yarn.executor.memoryOverhead.
Message template in the source: Container killed by YARN for exceeding physical memory limits. %s Consider boosting spark.executor.memoryOverhead.