dataengref errors

Home / Spark / Serialization

Spark: org.apache.spark.SparkException: Task not serializable

org.apache.spark.SparkExceptionTask not serializable

A function passed to a Spark transformation captures an object that cannot be serialized, so Spark cannot ship the closure to executors. The "Caused by: java.io.NotSerializableException" line underneath names the class.

Causes · Fixes · Source

Common causes

  1. The lambda references a field or method of the enclosing class, which pulls the whole non-serializable class into the closure.How to confirm: The serialization stack shows "field (class: YourClass, name: ...)" with your driver class.
  2. The closure captures a client object such as a database connection, HTTP client or logger.How to confirm: The NotSerializableException names a connection, client or logger class.

How to fix it

Copy the needed field into a local variable before the lambda, so only that value is captured.

val factor = this.factor
rdd.map(x => x * factor)

Create non-serializable clients on the executors, for example inside mapPartitions or foreachPartition.

df.foreachPartition { rows =>
  val conn = createConnection()
  rows.foreach(r => write(conn, r))
  conn.close()
}

Mark helper classes Serializable, or put shared functions in a Scala object instead of a class.

Where the message comes from

References

Last checked: