PySpark / Performance Fig_01  ·  Lesson 01 / 14
Fig_01 · Spark architecture · v1.0 OSS Spark + Azure Databricks

Spark architecture

The execution model · driver, executors, the shuffle

One line of PySpark becomes a fleet of tasks across a cluster: the driver builds a plan, an action fires a job, shuffle boundaries cut it into stages, and one task runs per partition. Get this one picture and every later lesson clicks into place.

Spark 3.2+ / DBR LTS shuffle.partitions 200 1 driver · N executors verified Jun 2026

01 · What it is

A small society of processes

A Spark application is one program — your notebook or job — that runs as three kinds of process working together:

Two more choices frame the picture: where you launch from — the edge / gateway node — and where the driver runs — the deployment mode (client vs cluster). The deep dive unpacks each; for now just hold the shape.

The one rule to remember Spark is lazy. Transformations only describe work; an action fires a job; the job is cut at shuffle boundaries into stages; each stage runs as tasks — one per partition. Job → stage → task → partition is how you read any Spark UI.
Driver vs executors

The one driver plans and schedules the work; the N executors run the tasks and hold the data.

Narrow — no shuffle

filter, select, withColumn stay in one stage; no data crosses the network.

Wide — shuffle

groupBy, join, distinct redistribute rows across the network — a new, costly stage.

02 · Why it matters

You can't tune what you can't see

Figure A · lazy vs action

Which line fires the job?

Click each line to classify it. Transformations are lazy — they only extend the plan; the action is the one line that submits a job. Watch the counter: it ticks only when you hit an action. (This isolates one idea — laziness. Where the work then runs is Figure B.)

lazy transformations vs the action

Tap a line to reveal whether it runs anything. Find the single line that triggers execution.

transformation (lazy — no job) action (fires a job)
df = spark.range(0, 1_000_000) df2 = df.filter("id % 2 = 0") df3 = df2.withColumn("bucket", df2.id % 10) df4 = df3.groupBy("bucket").count() total = df4.count() # <-- ? df4.write.format("noop").save() # <-- ?
0
jobs submitted so far
0
lazy lines found
Click any line to reveal whether it runs anything.

Simplified illustration — show() / take() also fire jobs; noop runs the full plan but discards output.

03 · How it works — deep dive

From one line of code to a fleet of tasks

The whole model is a chain: your code builds a plan, an action turns it into a job, shuffle boundaries cut it into stages, and each stage fans out into tasks. Each idea below is explained here once — later sections show it, verify it, and help you decide, but don't re-explain it.

3.1 · The three roles

analogy A restaurant kitchen — the driver is the head chef calling orders and plating the dish; the executors are the line cooks at their stations; the cluster manager is the owner deciding how many cooks you get.

use case A stage with 800 partitions on executors with 4 cores each needs 200 executor-cores to finish in one wave; fewer cores means more waves means longer wall-clock.

3.2 · Edge node & deployment mode — where the driver runs

3.3 · Lazy evaluation: transformations vs actions

3.4 · Narrow vs wide dependencies — and the shuffle

use case events.groupBy("user_id").count() shuffles all events so each user's rows land together — a cluster-killer when one user holds 90% of rows (skew, Lesson 08).

3.5 · The execution hierarchy: job → stage → task

This is the canonical enumeration — the comparison table and gotchas later point back here rather than re-listing it. Read it top-down: each level unfolds into the next.

LevelSplit ruleConcrete example
Action → JobEach action submits exactly one job.1 count() ⇒ 1 job
Job → StagesA new stage is cut at every shuffle (wide) boundary; narrow ops fuse into the same stage.1 shuffle ⇒ 2 stages
Stage → TasksEach stage runs one task per partition.200 partitions ⇒ 200 tasks
Tasks → coresTasks run on cores in waves (total tasks ÷ total cores).200 tasks ÷ 50 cores ⇒ 4 waves
Shuffle ≠ a plain scan's partitioning spark.sql.shuffle.partitions only sets the count after a shuffle. A plain scan's partition count comes from the input files; change that with repartition / coalesce, not this config. Conflating the two is the most common early mistake.

04 · The code — and how to verify it

Four moves, each with its proof

Every technique below is paired with how to prove it worked — the .explain() plan node or the Spark-UI signal. Apply, then verify; never assume.

Simple reading rule When you see a line of PySpark, ask one question first: "Does this line trigger an action (fire a job) or introduce a shuffle — or does it stay narrow and lazy?" An action or an Exchange is where the cost lives; a lazy narrow transformation is nearly free.

Prove lazy evaluation (transformations build, actions fire)

python# None of these run anything — they only extend the logical plan (LAZY).
df  = spark.range(0, 1_000_000)
df2 = df.filter("id % 2 = 0")                 # transformation — no job
df3 = df2.withColumn("bucket", df2.id % 10)   # transformation — no job

# VERIFY: Spark UI > Jobs tab is EMPTY right now. Nothing has executed.
count = df3.count()                          # ACTION — fires exactly one job
# VERIFY: Spark UI > Jobs tab now shows ONE new job for count().

See the plan & stage boundaries (narrow vs wide)

pythonfrom pyspark.sql.functions import col   # used by the col("id") snippets below

# Narrow-only: filter + withColumn fuse into ONE stage, no shuffle.
narrow = spark.range(1_000_000).filter("id > 10").withColumn("x", col("id") * 2)
narrow.explain(mode="formatted")
#   Project +- Filter +- Range     <- no Exchange = no shuffle = single stage

# Wide: groupBy forces a shuffle -> a stage boundary appears.
wide = spark.range(1_000_000).groupBy(col("id") % 100).count()
wide.explain(mode="formatted")
#   HashAggregate
#   +- Exchange hashpartitioning(...)   <- Exchange = the shuffle = a NEW stage (costly)
#      +- HashAggregate +- Range

Same idea in Spark SQL — the GROUP BY introduces the Exchange:

sqlEXPLAIN FORMATTED
SELECT id % 100 AS g, count(*) FROM range(1000000) GROUP BY id % 100;
-- Look for the `Exchange` node — that is the shuffle / stage boundary.

Inspect & control partitions (one task per partition)

pythondf = spark.range(1_000_000)
print(df.rdd.getNumPartitions())     # in-memory partitions => tasks per stage

shuffled = df.groupBy(col("id") % 7).count()
print(shuffled.rdd.getNumPartitions())  # -> 200 (post-shuffle count, OSS default)

# Demo only — set it small and watch tasks change (AQE may coalesce it, Lesson 05).
spark.conf.set("spark.sql.shuffle.partitions", 8)
# VERIFY: Spark UI > Stages tab — post-shuffle stage now shows 8 tasks, not 200.

Time a plan without pulling data to the driver

pythonimport time
# The "noop" sink runs the FULL job but writes nothing — time a plan WITHOUT
# a driver-OOM-risky collect() (Lesson 03).
t0 = time.time()
wide.write.format("noop").mode("overwrite").save()
print(round(time.time() - t0, 2), "s")
# VERIFY: Spark UI > SQL tab shows the completed query; Stages tab shows the
# Exchange's Shuffle Read/Write bytes — the cost of the wide op.

Figure B · the shuffle, step by step

See it run across the cluster

Everything above comes together here. This is events.groupBy("key").count() on a tiny cluster — 1 driver + 3 executors, with 9 records tagged A, B, or C scattered across the input partitions. Step through it with Prev / Play / Next and watch the shuffle physically move records across the network so every key lands on one executor — the wide-dependency boundary at the heart of this track.

cluster execution lab — the shuffle

Driver builds the plan; executors do the work. Use Prev / Play / Next to walk the stages.

Driver plan + schedule

idle

Executor 1 2 cores

Executor 2 2 cores

Executor 3 2 cores

step 1 / 6
dummy PySpark
# 9 events, key in {A,B,C}, 3 input partitionsevents = spark.read.parquet("/events")tagged = events.withColumn("key", key_udf) # narrowgrouped = tagged.groupBy("key") # wide → shuffleresult = grouped.count() # action → jobresult.show()
per-key result
key A → ·
key B → ·
key C → ·

Simplified illustration — real shuffles use spark.sql.shuffle.partitions (200 in OSS; auto on Databricks) post-shuffle partitions, not one per key.

05 · Summary table

Narrow vs wide — and where the driver runs

Read the first table by the operation you're about to call — it answers "does this shuffle?" at a glance. (The why is §3.4; this is the which.)

DimensionNarrow dependencyWide dependency (shuffle)
Examplesmap, filter, select, withColumn, uniongroupBy, join (non-broadcast), distinct, repartition
Data movementNone — same executorRows redistributed across the network
StageFused into the same stageForces a new stage (a barrier)
Plan nodeno ExchangeExchange (one per shuffle)
CostCheap (CPU only)Expensive — disk + network + serialize + wait
Deployment modeWhere the driver runsClient can disconnect?Best for
ClientSubmitting / edge node (Databricks: driver node)No — client death kills the appInteractive notebooks, REPL
ClusterInside the cluster (YARN ApplicationMaster)Yes — fire-and-forgetProduction batch / scheduled jobs

06 · Edge cases & limitations

Where it gets subtle

New nuance an interviewer probes — not a re-run of the hierarchy above:

Hard limits worth stating plainly:

07 · Common mistakes

Gotchas — the mistake and its one-line fix

MistakeWhy it hurtsBetter move
Thinking transformations runOnly actions execute; a "slow filter" is really the action that finally ran the whole chain.Time the action and read the plan, not the transformation.
Assuming more executors = fasterParallelism is capped by the partition count; 1000 cores can't help a stage with 8 partitions.Match partitions to cores before adding executors.
Ignoring the ExchangeEvery Exchange is a shuffle you're paying for.Ask which Exchange nodes are present and whether one can be removed.
collect() "just to look" at a big DataFramePulls every row to the driver and risks driver OOM (Lesson 03).Use take(n), show(), or a noop write.
Mixing up the cluster manager and the driverThe manager only allocates machines — it does not schedule tasks.Remember the driver schedules tasks; the manager just hands out machines.

08 · References

Sources