01 · What it is
Read the plan as an assembly line
A Spark query plan is the path from what you wrote to what executors run. The order matters: Spark first understands the query, then improves the logical plan, then chooses a physical strategy, then executes it.
explain(True) shows how Spark understood and optimized your query. The bottom, Physical Plan, shows what Spark will actually run.02 · Interactive pipeline
Step through the image left to right
Click each stage in the same order as the diagram.
customers.filter(col("city") == "boston").select("cust_id", "name")03 · How to read explain
The four sections Spark prints
| Section | Plain meaning | What to notice |
|---|---|---|
| Parsed Logical Plan | Spark parsed your code but names may still be unresolved | Does this match what you wrote? |
| Analyzed Logical Plan | Catalog/schema resolved columns, tables, functions, types | Wrong column/type errors fail here |
| Optimized Logical Plan | Catalyst removes unnecessary work | Column pruning, pushed filters, collapsed projections |
| Physical Plan | Executable operators Spark selected | Exchange, joins, aggregates, scans, AQE |
df.explain(True)
# == Parsed Logical Plan ==
# == Analyzed Logical Plan ==
# == Optimized Logical Plan ==
# == Physical Plan ==04 · Notebook examples
Narrow vs wide plans
analogyNarrow operations are like editing each page in place. Wide operations are like collecting pages from every desk and redistributing them by topic.
| Example | Plan clue | Meaning |
|---|---|---|
| filter / select / withColumn | Filter, Project, no Exchange | Local per-partition work |
repartition(24) | Exchange RoundRobinPartitioning(24) | Full shuffle to create new partitions |
coalesce(5) | Coalesce 5 | Merge partitions without full shuffle |
| join with broadcast disabled | SortMergeJoin + two Exchange nodes | Both sides shuffle by join key |
groupBy().count() | HashAggregate → Exchange → HashAggregate | Partial aggregate, shuffle, final aggregate |
05 · Physical node checklist
Start with expensive nodes
| Node | Plain meaning | Next check |
|---|---|---|
FileScan parquet | Read files | ReadSchema, PushedFilters, PartitionFilters |
Exchange | Shuffle / repartition | Spark UI shuffle read/write |
Coalesce | Reduce partitions cheaply | Possible imbalance |
SortMergeJoin | Shuffle, sort, merge | Join size and skew |
HashAggregate | Aggregate by key | Partial vs final aggregate |
AQEShuffleRead | AQE changed shuffle read | Final adaptive plan after action |
06 · Pushdown note
Why a filter can appear twice
If Spark pushes a filter into Parquet/Delta, you may still see a Filter above the scan. That is not necessarily duplicate waste. Spark keeps a physical filter as a correctness check because not every source can fully apply every pushed predicate.
customers.filter(col("age").cast("int") > 50).explain(True)
# Casts and complex expressions may limit pushdown.
# Check PushedFilters and the physical Filter.07 · Layer names
Catalyst chooses; engines execute
| Term | Where it fits | Simple meaning |
|---|---|---|
| Catalyst | Logical + physical planning | Optimizes and chooses query plans |
| Cost model | Physical planning | Picks from candidate physical plans |
| AQE | Runtime planning | Revises after real shuffle stats |
| Tungsten | Execution internals | Compact memory and codegen |
| Photon | Databricks execution engine | Native execution for compatible operators |