Home / Spark / Shuffle and joins
Spark: Could not execute broadcast in 300 secs
Could not execute broadcast in 300 secs. You can increase the timeout for broadcasts via spark.sql.broadcastTimeout or disable broadcast join by setting spark.sql.autoBroadcastJoinThreshold to -1 or remove the broadcast hint if it exists in your code.A broadcast join could not build and send its small side within spark.sql.broadcastTimeout (300 seconds by default). Either the "small" table is not small, or the query that produces it is slow. Older releases surface the same problem as "Futures timed out after [300 seconds]".
Common causes
- The broadcast side is larger than the statistics suggested, for example after a filter or join that the optimizer misestimated.How to confirm: In the SQL tab, the BroadcastExchange node shows a large "data size" or a long "time to build".
- The subquery that produces the broadcast side is expensive and takes most of the timeout.How to confirm: The job that computes the broadcast side is long-running in the Jobs tab.
- A BROADCAST hint forces a large table to be broadcast.How to confirm: The query contains /*+ BROADCAST(t) */ or broadcast(df).
How to fix it
Let Spark choose the join: remove the hint, or disable automatic broadcast joins for this query.
| Setting | Default | Note |
|---|---|---|
spark.sql.autoBroadcastJoinThreshold | 10MB | -1 disables broadcast joins |
If the broadcast is correct but slow to build, raise the timeout.
| Setting | Default | Note |
|---|---|---|
spark.sql.broadcastTimeout | 300 (seconds) |
Where the message comes from
error-conditions.jsoninapache/sparkatv4.1.3- Message checked in Spark 4.1.3 (error condition _LEGACY_ERROR_TEMP_2097)
Message template in the source: Could not execute broadcast in <timeout> secs. You can increase the timeout for broadcasts via <broadcastTimeout> or disable broadcast join by setting <autoBroadcastJoinThreshold> to -1 or remove the broadcast hint if it exists in your code.