Spark UI and query plan debugging · PySpark Performance

01 · What it is

Plan first, metrics second

df.explain(mode="formatted") tells you the operators Spark planned. The Spark UI tells you how those operators behaved at runtime: shuffle, spill, skew, GC, retries, cache, and executor health.

The one rule to rememberExplain shows the shape; Spark UI shows the cost. Use both before changing code, memory, or cluster size.

Plan clue

Exchange, join strategy, scan filters, cache scans, and AQE nodes.

Runtime clue

Task spread, shuffle read/write, spill, GC time, failed tasks, executor health.

Healthy pattern

The plan changed and the matching UI metric improved after your fix.

02 · How it works

The diagnostic loop

analogyThe plan is a blueprint; the UI is the construction bill. A big room on the blueprint is not bad by itself, but the bill tells you which room consumed all the time and material.

query.explain(mode="formatted")
# Look for: Exchange, BroadcastHashJoin, SortMergeJoin, HashAggregate,
# InMemoryTableScan, AdaptiveSparkPlan, AQEShuffleRead.

spark.sparkContext.setJobGroup("L12: slow join", "L12: slow join", True)
query.write.format("noop").mode("overwrite").save()
# VERIFY: Spark UI -> SQL tab for the DAG; Stages tab for task metrics.

Tab	Use it for	Signal
SQL/DataFrame	Connect plan nodes to the runtime DAG	Exchange, BroadcastExchange, AQEShuffleRead
Stages	Find the expensive stage	Max vs Median task time, shuffle, spill, GC
Executors	Separate cluster issue from one bad node	Failed tasks, task time, GC, driver row
Storage	Confirm cache actually materialized	Cached partitions, memory, disk, eviction
Environment	Confirm configs really applied	AQE, shuffle partitions, memory flags

03 · Interactive diagnosis

Pick the signal, then choose the next check

Spark UI signal mapper

Choose the symptom you see in the UI and watch the likely root cause and next evidence update.

Shuffle-heavy plan

likely cause

SQL DAG

next check

Look for Exchange nodes and high Shuffle Read/Write. Try pruning, broadcast, bucketing, or AQE before sizing memory.

04 · Runbook

If you see X, check Y

Symptom	Check first	Likely fix
Slow join	Join node + Exchange count	Broadcast, AQE, bucketing
One straggler	Max vs Median + shuffle size	AQE skew split or salting
Slow but succeeds	Spill memory/disk	Smaller partitions or memory
High GC	GC Time and executor logs	Less object churn, cache, G1/off-heap
Driver OOM	Driver row and action type	Avoid collect, reduce metadata

05 · Uses and gotchas

Use evidence, not guesses

Use it for slow joins, skew, spill, GC, failed tasks, and cache validation.
Edge case: AQE final plans differ from initial plans; inspect after an action.
Limitation: UI history can expire, and logs may still be needed for executor failures.
Gotcha: benchmarking with collect() adds driver pressure. Use a real write or noop sink when you want full-plan runtime.