Hive: Number of dynamic partitions created is N, which is more than 1000
org.apache.hadoop.hive.ql.metadata.HiveExceptionNumber of dynamic partitions created is 1460, which is more than 1000. To solve this try to set hive.exec.max.dynamic.partitions to at least 1460.A dynamic-partition INSERT would create more partitions than hive.exec.max.dynamic.partitions (1000 by default). Before raising the limit, check whether the partition column is right: thousands of partitions from one insert often come from the wrong column order in the SELECT.
Common causes
- The partition columns are not the last columns of the SELECT, so a high-cardinality column such as an ID is used as the partition value.How to confirm: Run the SELECT alone and look at the values in the last columns; they should be the partition values.
- The insert genuinely spans many partitions, for example a backfill of several years of daily data.How to confirm: Count distinct partition values in the source: SELECT COUNT(DISTINCT dt) FROM source.
How to fix it
Put the partition columns last in the SELECT, in the same order as in PARTITION (...).
For real backfills, raise the total and per-node limits, or load in smaller date ranges.
| Setting | Default | Note |
|---|---|---|
hive.exec.max.dynamic.partitions | 1000 | |
hive.exec.max.dynamic.partitions.pernode | 100 |
Where the message comes from
Utilities.javainapache/hiveatrel/release-4.2.1- Message checked in Hive 4.2.1
Message template in the source: Number of dynamic partitions created is %d, which is more than %d. To solve this try to set hive.exec.max.dynamic.partitions to at least %d.