Home / Spark / Resources and memory
Spark: Total size of serialized results is bigger than spark.driver.maxResultSize
org.apache.spark.SparkExceptionTotal size of serialized results of 1543 tasks (1024.3 MiB) is bigger than spark.driver.maxResultSize (1024.0 MiB)The tasks of one job sent more result data back to the driver than spark.driver.maxResultSize (1g by default) allows, so the job was aborted. collect(), toPandas() and large broadcast builds are the usual sources.
Common causes
- The code collects a large DataFrame to the driver with collect(), toPandas() or take() with a large n.How to confirm: The failing job in the Spark UI is triggered by a collect or toPandas call.
- A broadcast join builds a large table on the driver.How to confirm: The failing job belongs to a BroadcastExchange in the SQL tab plan.
- Many tasks each return a modest result, which adds up.How to confirm: The message shows a high task count with a moderate average size.
How to fix it
Avoid bringing data to the driver: write results to storage, or aggregate before collecting.
If the data really must come back, raise the limit and the driver memory together; the limit protects the driver from OOM.
| Setting | Default | Note |
|---|---|---|
spark.driver.maxResultSize | 1g | 0 means unlimited |
spark.driver.memory | 1g |
Where the message comes from
TaskSetManager.scalainapache/sparkatv4.1.3- Message checked in Spark 4.1.3
Message template in the source: Total size of serialized results of %d tasks (%s) is bigger than spark.driver.maxResultSize (%s)