Driver memory

01 · What it is

One node, one heap, one job to keep it small

Every Spark application has exactly one driver — the JVM that runs your notebook, holds the SparkSession, builds the plan, and schedules work onto the executors. Its heap (spark.driver.memory, default 1g) is fixed. A driver OOM is simply what happens when something you asked Spark to bring back to the driver is too big to fit.

analogy The driver is the air-traffic controller in the tower: it doesn't fly the planes — the executors do — it just tracks every flight. Order all the cargo delivered to the tower and you crush it, no matter how many planes you have.

A handful of things share that one heap — action results, the build side of a broadcast, and the plan/scheduling metadata. The deep dive below breaks down each and how it fails; for now just hold the shape of the problem:

02 · Why it matters

The driver is the one thing you can't lose

• It's a single point of failure. A slow executor degrades one task; a dead driver kills the whole application — every executor's work is thrown away.
• It fails for non-obvious reasons. The data on the executors can be perfectly healthy while the driver dies — so driver problems are the ones that look mysterious.
• It's a classic interview probe: "A job ran fine for months, then started failing — with no growth in the executor-side data. Where do you look?" The answer is usually the driver: a result set, a broadcast, or a partition count that crept over the limit. The how is the next section.

03 · How it works — deep dive

What fills the driver heap, and how it tips over

Everything a driver OOM comes down to is data arriving at one fixed heap faster than it can hold. So the useful way to understand it is: (1) what legitimately lives on that heap, (2) the guardrail that's supposed to catch overflow, and (3) the specific paths that defeat it. Each idea below is explained here once — later sections show it, verify it, and help you decide, but don't re-explain it.

3.1 · What shares the one heap

Use this as the driver-memory map: if something lands in one of these rows and grows too large, the driver is the node under pressure.

3.2 · spark.driver.maxResultSize — the guardrail

use case A tripwire so a stray collect() aborts cleanly with a clear error, instead of silently thrashing the heap into GC death.

• What it is (doc-quoted). "Limit of total size of serialized results of all partitions for each Spark action (e.g. collect). Jobs will be aborted if the total size is above this limit. Having a high limit may cause out-of-memory errors in driver." Default 1g.
• The mechanism. As task results stream back, Spark sums their serialized sizes; the instant the running total crosses the limit, the job is aborted — a pre-emptive stop, before the heap is necessarily exhausted.
• Why it's worth having. It turns a creeping, GC-thrashing driver OOM (slow, hard to diagnose) into a fast, explicit failure with an actionable message.
• The trap (Rung 3). Raising it — or setting 0 for "unlimited" — removes the tripwire, so a too-big collect() sails past the clean abort into a real OutOfMemoryError. Raise it only after you've sized spark.driver.memory to match.

3.3 · The four paths to a driver OOM

Every real-world driver OOM is one of these four. This is the canonical list — the comparison table and gotchas later point back here rather than re-explaining.

Figure A · the tipping point

Watch the heap cross the line

Drag how many rows you collect() and watch the serialized result climb toward the maxResultSize line. This figure isolates one thing: where the threshold is — the clean abort, and what raising the cap turns it into. (Where the data physically travels is Figure B.)

04 · The code — and how to verify it

Four moves, each with its proof

Every technique below is paired with how to prove it worked — the plan node or Spark-UI signal. Apply, then verify; never assume.

Replace collect() with a write (or a bounded pull)

python# NAIVE: pulls EVERY row of a fact table into the 1g driver heap → abort or OOM.
rows = big_df.collect()                    # all partitions serialized into one heap
pdf  = big_df.toPandas()                   # same hazard, plus the Python process

# RIGHT: keep the data on the executors — write it out…
big_df.write.mode("overwrite").saveAsTable("main.pyspark_perf.result")

# …or RUN the job for timing without returning rows, via the noop sink:
big_df.write.format("noop").mode("overwrite").save()

# Need a few rows locally? Pull a BOUNDED set, never the whole thing:
sample = big_df.limit(1000).collect()      # at most 1000 rows reach the driver

# VERIFY — driver log on a too-big collect() shows:
#   "Total size of serialized results ... is bigger than spark.driver.maxResultSize"

Aggregate on the cluster, then collect the small summary

pythonfrom pyspark.sql import functions as F

total_bad = sum(r["amount"] for r in big_df.collect())   # pulls every row — hazard
total_ok  = big_df.agg(F.sum("amount")).collect()[0][0]   # 1 row to the driver

# VERIFY — .explain() shows HashAggregate (partial + final) on the cluster;
# the final exchange is tiny, not the whole table.

Guard against a mis-sized broadcast (driver-side build)

pythonfrom pyspark.sql.functions import broadcast

risky = fact.join(broadcast(too_big_dim), "k")   # small side built on the driver → OOM
ok    = fact.join(small_dim, "k")                 # let the planner/AQE broadcast only if it fits

spark.conf.set("spark.sql.autoBroadcastJoinThreshold", 10*1024*1024)  # 10 MB (OSS default)

# VERIFY — fact.join(...).explain(mode="formatted"): a BroadcastExchange on the SMALL
# side is fine; on a multi-GB side it's the warning sign. The OOM, if it comes, is on
# the DRIVER log, not an executor.

Size the two limits together, deliberately

pythonprint(spark.conf.get("spark.driver.memory"))         # '1g' — the heap (launch-time)
print(spark.conf.get("spark.driver.maxResultSize"))  # '1g' — the guardrail (runtime-settable)

spark.conf.set("spark.driver.maxResultSize", "2g")   # raise the guardrail ONLY with a matching heap
# spark.driver.memory is a launch-time JVM flag → set it in the cluster config + restart:
#   spark.driver.memory 8g

# VERIFY — Spark UI → Executors tab: the single 'driver' row shows memory + GC time.
# Rising driver GC time during a collect() = the heap is under pressure.

Figure B · where the rows go

See it run across the cluster

This figure isolates a different thing from Figure A: not the threshold, but where the data physically lands. Step a collect() across 1 driver + 3 executors and watch results pile onto the one heap — then flip to df.write and watch the driver stay flat. Use Prev / Play / Next.

05 · Decision table

Which action, and what it costs the driver

Read this by the action you're about to call — it answers "will this touch the driver?" at a glance. (The why is §3; this is the which.)

06 · Edge cases & limitations

Where it gets subtle

New nuance an interviewer probes — not a re-run of the four paths:

• The broadcast that's almost too big. A dimension at 8 MB today, 80 MB next quarter, silently starts OOMing the driver during the build. Cap the threshold; let AQE decide at runtime rather than forcing broadcast().
• toPandas() hits two meters at once. Every row lands in the JVM heap and is converted in the Python driver process (overhead) — it can OOM the container even when the JVM heap alone looks fine.
• Raising the cap without raising the heap. Removing the guardrail doesn't add capacity — it just converts a clean abort into a slower, harder-to-diagnose OOM.

Hard limits worth stating plainly:

• The driver does not scale out. There is exactly one, one fixed heap — you can't "add driver nodes." Above the heap the only moves are bring less back or a bigger driver node.
• spark.driver.memory is launch-time. It's a JVM flag set when the driver starts — set it in the cluster Spark config (a restart), not spark.conf.set(...). Only maxResultSize is runtime-settable.
• Doc-grounding caveat. Only path (a) — the collect()/large-result route via maxResultSize — is verbatim in the docs; paths (b)–(d) are well-established operational practice, stated as such.

07 · Common mistakes

Gotchas — the mistake and its one-line fix

08 · References

Sources

• Apache Spark — Configuration (driver.memory, maxResultSize, memoryOverhead)
• Apache Spark — Cluster overview (driver, executors, what the driver does)
• Apache Spark — Tuning guide (memory management, serialized results)
• Apache Spark — SQL Performance Tuning (broadcast join, driver-side build)
• Azure Databricks — Compute configuration (the driver node)