How to use this track
Read it in order — it's one story, not a menu
The track follows a Spark job from how it runs to how you make it fast: read the plan, avoid the shuffle, fix skew, reuse work, tune memory and GC, then debug and size production jobs. Parts A → B → C → D build on each other, so the intended path is 01 through 14. Each page stands alone if you need to jump straight to a topic.
→ Full learning plan (suggested order, timings & self-check questions)
.explain()) and the Spark UI first. Then, in order: read less (partition pruning / DPP, column pruning) → avoid or repair the shuffle (broadcast the small side, bucket repeated joins, let AQE coalesce) → fix skew (AQE skew join, then salting) → reuse work (cache only reused DataFrames) → size memory and GC last. Always verify the change in .explain() / the Spark UI — never assume.
The lessons
Fourteen lessons, four parts
Part A · The engine
How a PySpark line of code becomes work on a cluster, the join strategies that move (or save) the most data, and where memory goes on the driver and executors.
Spark architecture and the execution model
Driver, executors, and cluster manager; job → stage → task; lazy evaluation, narrow vs wide dependencies, and the shuffle.
Open lessonJoins: Sort-Merge vs Shuffle-Hash vs Broadcast
Three strategies and how Spark chooses; broadcast() to skip the shuffle; reading the join plan in the Spark UI.
Driver memory and driver OOM
What lives on the driver — collect(), the broadcast build, plan metadata — plus maxResultSize and why the driver OOMs.
Executor memory: unified model, spill and OOM
Reserved 300 MiB, the storage⇄execution borrow and eviction asymmetry, off-heap and PySpark-worker memory, spilling, and executor OOM.
Open lessonPart B · Runtime adaptivity and reuse
The plan that rewrites itself at runtime, how to stop recomputing the same DataFrame, and how to read far fewer files.
Adaptive Query Execution (AQE)
Coalesce post-shuffle partitions, split skewed ones, and switch sort-merge to broadcast at runtime using real shuffle stats.
Open lessonCache and persist
cache() vs persist() and the storage-level matrix; lazy materialization; and unpersist() when you're done.
Partition pruning and dynamic partition pruning
On-disk vs in-memory partitions; static pruning on a partition-column filter; DPP pruning fact partitions from a join at runtime.
Open lessonPart C · Skew, shared variables and last-mile tuning
The recurring villain (skew) and its fixes, the two shared-variable types, and the JVM-level levers — GC and bucketing.
Data skew: salting and SQL hints
What skew is and how to spot it; salting aggregations and joins; join and partitioning hints to steer the planner.
Open lessonBroadcast variables and accumulators
Read-only broadcast variables shipped once per executor, write-only accumulators the driver reads, and the exactly-once caveat.
Open lessonGarbage-collection tuning
Young/old generations, minor vs major GC, stop-the-world pauses, and cutting GC time with serialized caching, fewer objects, and G1GC.
Open lessonBucketing to eliminate the shuffle
bucketBy(n,col).sortBy(col).saveAsTable(); equal buckets on the join key mean no Exchange; bucketing vs partitioning.
Part D · Debugging and production readiness
The practical FDE layer: read Spark UI evidence, size partitions and clusters, and explain the optimizer and execution-engine terms behind physical plans.
Spark UI and query plan debugging
Use explain(), SQL DAGs, Stages, Executors, Storage, and Environment tabs to diagnose slow joins, skew, spill, GC, OOM, and cache reuse.
Partition, shuffle and cluster sizing
Task waves, executor cores, repartition() vs coalesce(), shuffle partition upper bounds, memory overhead, and when workers help.
Catalyst, Tungsten and physical plan nodes
Logical vs physical plans, optimizer-visible DataFrame code, common plan nodes, whole-stage codegen, and AQE vs execution-engine layers.
Open lessonField guide
Real-world scenarios FDEs face
Beyond the lessons: the recurring Databricks performance calls you make on customer workloads — a slipping SLA, an autoscaling job that plateaus, a driver that dies on a "small" result — each with the diagnosis and the fix, cross-linked to the lesson that explains the mechanism.