01 · What it is
Partitions create tasks; cores run tasks
A stage runs one task per partition. Executor cores are task slots. If a stage has fewer partitions than cores, the cluster waits idle; if it has millions of tiny partitions, scheduling overhead dominates.
The one rule to rememberPartition count controls parallelism; partition size controls memory pressure. Tune for enough full task waves, not the highest possible partition count.
02 · Interactive sizing
Estimate task waves
Partition wave calculator
Move the sliders and watch how partitions, cores, and average partition size change the stage shape.
7
task waves
307 MB
avg partition
192
AQE target partitions
03 · Key levers
Choose the smallest lever
analogyExecutor cores are checkout counters; partitions are customers. Too few customers leaves counters idle; too many tiny customers makes line management the bottleneck.
| Lever | Use it when | Cost |
|---|---|---|
coalesce(n) | Reduce partitions cheaply for small output | Can preserve imbalance |
repartition(n, col) | Need a new balanced key distribution | Full shuffle |
| AQE coalesce | Post-shuffle partitions are tiny | Needs shuffle stats |
| More workers | Enough balanced tasks exist | Does not fix skew or bad plans |
| More overhead | PySpark/Pandas UDF memory pressure | Cluster resource increase |
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.shuffle.partitions", 400) # upper bound before AQE
agg = events.groupBy("customer_id").sum("amount")
agg.explain(mode="formatted")
# VERIFY after action: AdaptiveSparkPlan + AQEShuffleRead coalesced04 · Uses and gotchas
Sizing does not replace better plans
- Use it to avoid idle cores, tiny tasks, huge spilling partitions, and tiny output files.
- Edge case: one huge partition is skew, not a global partition-count problem.
- Limitation: AQE coalesces post-shuffle partitions; it does not compact existing input files.
- Gotcha: adding workers to a 20-partition stage will not use 200 cores.