Databricks · One-Pager · Performance

Spark performance tuning

The levers that decide how fast a Spark job runs: let AQE re-plan at runtime, pick the right join, and stop reading data you don't need.

AQE on by default broadcast threshold 10 MB Verified June 2026 docs

In one line

Spark performance tuning is removing the work the engine doesn't need to do — by skipping files, shuffling less, joining smart, and letting Adaptive Query Execution re-optimize the plan from real runtime stats.

Why it matters

Most slow Spark jobs are not CPU-bound — they are data-movement-bound: oversized shuffles, one straggler task on skewed data, or a sort-merge join where a broadcast would have done. Tuning targets those, and AQE handles many of them automatically once you understand what it is doing.

Core concepts

Adaptive Query Execution

Re-optimization during execution using runtime stats: coalesces shuffle partitions, switches sort-merge to broadcast, and splits skewed partitions. On by default.

Broadcast vs sort-merge join

Broadcast ships the small side to every executor (no shuffle); sort-merge shuffles and sorts both sides. Auto-broadcast triggers under autoBroadcastJoinThreshold (10 MB).

Data skew

A few keys hold most rows, so one task runs far longer than the rest. AQE's skew join splits and replicates skewed partitions into evenly sized tasks.

Shuffle & spill

A shuffle redistributes data across the network between stages; when a task's data exceeds memory it spills to disk — the signal to repartition or shrink the shuffle.

Caching: disk vs Spark

Disk cache auto-caches remote Parquet on local SSD on the first read; cache()/persist() manually pin any DataFrame in memory. Prefer disk cache.

Pruning & data skipping

Data skipping uses per-file min/max stats (first 32 cols) to skip files; dynamic file pruning pushes a join's filter down to skip files at runtime.

How AQE re-plans a join

Click a stage

1 · Static plan 2 · Shuffle + stats 3 · Coalesce / switch 4 · Skew split

Static plan — Catalyst builds an initial physical plan from compile-time estimates, which can be wrong before any data moves.

Minimal example

PySparkfrom pyspark.sql.functions import broadcast

orders = spark.table("orders")        # large fact table
regions = spark.table("regions")      # small dimension

# force a broadcast hash join, skipping the shuffle on the small side
joined = orders.join(broadcast(regions), "region_id")

# SQL equivalent: SELECT /*+ BROADCAST(regions) */ ... FROM orders JOIN regions ...

Gotchas & limits

Broadcasting a side that isn't actually small risks driver/executor OOM — the 10 MB autoBroadcastJoinThreshold default exists for a reason; raise it deliberately.
A BROADCAST hint is a suggestion, not a guarantee, and hint precedence is BROADCAST > MERGE > SHUFFLE_HASH > SHUFFLE_REPLICATE_NL.
AQE skew handling triggers only when a partition exceeds both skewedPartitionThresholdInBytes (256 MB) and 5× the median — tiny-but-uneven skew slips through.
cache() is lazy and memory-bound; over-caching evicts other blocks and can hurt. On Databricks prefer the automatic disk cache for Parquet/Delta.

Interview soundbites

Say this out loud

"AQE re-optimizes the plan mid-flight from shuffle stats — it coalesces small partitions, flips sort-merge to broadcast when a side turns out small, and splits skewed partitions. It's on by default."
"Broadcast join avoids the shuffle by shipping the small table to every executor; sort-merge shuffles and sorts both sides. Spark auto-broadcasts below 10 MB, or I hint broadcast(df)."
"For skew I lean on AQE's skew join; for I/O I lean on data skipping and dynamic file pruning so Spark reads fewer files in the first place."

References

Verified against docs.databricks.com · June 2026