Home / Spark / Connectors and classpath

Spark: [UNABLE_TO_INFER_SCHEMA] Unable to infer schema for Parquet. It must be specified manually

Spark found no data files to read the schema from. The path is empty, contains only metadata or marker files, or points to the wrong directory level.

Causes · Fixes · Source

Common causes

1. The directory exists but has no data files yet, for example an upstream job wrote nothing for that date.How to confirm: List the path; it is empty or contains only _SUCCESS or hidden files.
2. The path points one level too high or too low in a partitioned layout.How to confirm: Data files sit in subdirectories such as dt=2025-01-01/ that the path does not reach.

How to fix it

spark.read.schema("id BIGINT, amount DOUBLE").parquet(path)

Where the message comes from

• error-conditions.json in apache/spark at v4.1.3
• Message checked in Spark 4.1.3 (error condition UNABLE_TO_INFER_SCHEMA)

Message template in the source: Unable to infer schema for <format>. It must be specified manually.

References

• spark.apache.org/docs/4.1.3/sql-data-sources-parquet.html