Garbage collection

01 · What it is

A pause you didn't schedule

Garbage collection (GC) is the JVM automatically reclaiming the memory of objects your program no longer references — in Java/Scala you never call free(), the collector does it for you. On a Spark executor, that convenience shows up as one thing you can measure: pauses.

analogy GC is the cleaning crew at a restaurant. While they sweep the whole floor, every diner steps outside and waits — and the more plates and crumbs (objects) on the floor, the longer everyone waits at the door. A Spark executor is one restaurant; its task threads are the diners.

The deep dive below breaks down the generations, the collection types, and the levers one at a time; for now just hold the shape of the problem — a pause you didn't ask for, paid for in idle CPU.

02 · Why it matters

One executor's pause is the whole stage's problem

• A GC pause is wasted cluster time. During a stop-the-world collection no task on that executor makes progress — you are paying for CPUs that sit idle while the JVM tidies up.
• A stage waits on its slowest task. GC runs per executor JVM, so a long pause on one executor stalls its tasks while the rest of the cluster idles at the stage boundary. One GC-thrashing executor becomes a straggler that dominates the stage's wall-clock.
• Heavy caching is the usual trigger. Cached blocks are long-lived, so they pile into the Old generation; a full Old gen forces frequent, expensive full GCs — the "I cached everything and now it's slower" surprise.
• It's a classic interview probe: "The job slowed down and you see high GC time — what's happening, and what do you change?" The answer is a chain: too many objects / Old gen too full → stop-the-world pauses → executor stalls → tune cache shape, object shape, spark.memory.fraction, or the collector. The how is the next section.

03 · How it works — deep dive

The generations, the collections, and the pause

Everything about GC comes down to the collector walking live objects while the world stops. So the useful way to understand it is: (1) how the heap is split so most collections stay cheap, (2) the two kinds of collection and the pause they share, and (3) how the executor turns a local pause into a stage-wide stall. Each idea below is explained here once — later sections show it, verify it, and help you decide, but don't re-explain it.

3.1 · The heap generations — Young and Old

• The split. The executor heap is divided into a Young generation — Eden plus two Survivor spaces (S0/S1) — and an Old generation. New objects are allocated in Eden.
• How objects move. When Eden fills, a collection copies the still-live objects into a Survivor space; objects that survive enough of these collections are promoted to the Old generation.
• Why two generations. Most objects die young (the "weak generational hypothesis"), so collecting the small, churny Young space frequently is cheap; the long-lived Old space needs the expensive collection only rarely.
• Where Spark's data lands. Cached blocks and broadcast data are long-lived → they end up in Old. Cache too much and Old fills, which is the direct link to GC pain in 3.3.

analogy Eden is the prep counter (constant fast turnover); the Old gen is the walk-in freezer (things that stick around). You wipe the counter constantly and cheaply; you only defrost the freezer occasionally — and defrosting shuts the kitchen.

3.2 · Minor vs full GC — and the stop-the-world pause

• Minor GC collects the Young gen only — frequent and cheap, because Eden is small and most of it is already dead.
• Major / full GC collects the Old gen (often the whole heap) — rare and expensive, because it walks far more live objects.
• Stop-the-world. During either collection the JVM suspends every application thread. On an executor that freezes all task threads for the duration — a minor GC is a short blip; a full GC can be a multi-second stall that looks like a hung executor.
• The tuning goal. You can't eliminate GC, only make it shorter and rarer: fewer objects (both collections get cheaper) and a less-full Old gen (fewer full GCs). Every lever in §04 targets one of those two.

3.3 · The executor's role — a local pause, a global stall

use case One executor caches more than its share, its Old gen stays near-full, and it spends seconds in full GC every few tasks. Its tasks finish late, the stage can't complete until that straggler does, and the job's wall-clock balloons — even though the other executors were idle and healthy.

• GC is per-JVM. Each executor is its own JVM with its own heap, so GC runs independently inside each one. There is no cluster-wide GC.
• A pause stalls only that executor — but the stage waits for it. Because a stage completes only when its slowest task finishes, a GC-bound executor becomes the straggler that holds up everyone (§3.2's full GC is what creates it).
• What loads the Old gen. Cached blocks (especially deserialized) and broadcast data are long-lived → more Old-gen pressure → more full GCs. This is the concrete mechanism behind "caching too much" hurting.

Figure A · a pause across the cluster

Watch one executor freeze the stage

Three executors, each its own JVM heap, run tasks for one stage. We zoom in on Executor 1's heap — Eden (Young) and Old gen — and watch objects pile up, a minor GC clear Eden, then a full GC freeze every task thread on that executor while Executors 2 and 3 keep going. Watch the red frozen chips on Executor 1 and the driver's "waiting" note — the stage can't finish until the straggler resumes. Use Prev / Play / Next.

04 · The code — and how to verify it

Four levers, each with its proof

Every technique below is paired with how to prove it worked — the Spark-UI GC Time signal or the executor's GC log. Apply, then verify; never assume.

Turn on detailed GC logging first

python# Read when the executor JVM STARTS — set it in the cluster's Spark config, not mid-session.
spark.conf.set(
    "spark.executor.extraJavaOptions",
    "-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
)

# VERIFY — Spark UI → Executors → stdout. You'll see one line per collection:
#   [GC (Allocation Failure) ...]   <- minor GC (Young)
#   [Full GC (Ergonomics) ...]      <- full GC; many of these = Old gen too full

Lever 1 · Serialized caching — a JVM/Scala-side win (not a PySpark switch)

scala// Scala / Java (JVM-side) — where the serialized-cache GC win is real.
import org.apache.spark.storage.StorageLevel

// Deserialized JVM cache: one Java object per field per row →
// millions of long-lived objects in the Old gen → heavy full GC.
hotDf.persist(StorageLevel.MEMORY_AND_DISK)

// Serialized JVM cache: docs say it stores "only one object (a byte array)
// per RDD partition" → orders of magnitude fewer objects to track.
hotDf.persist(StorageLevel.MEMORY_AND_DISK_SER)
hotDf.count()   // materialize the cache

// VERIFY — Stages tab: GC Time drops on JVM caches; fewer "Full GC" lines in stdout.

Lever 2 · Lower spark.memory.fraction when the Old gen is full

python# Unified region = spark.memory.fraction × (heap − 300 MiB). Default 0.6.
# If cached blocks keep the Old gen near-full and you see constant full GCs,
# give the cache LESS heap so fewer long-lived objects are pinned.
spark.conf.set("spark.memory.fraction", "0.4")   # shrink cache/exec → less Old-gen pressure

# Docs verbatim: "better to cache fewer objects than to slow down task execution."
# VERIFY — Stages tab: compare GC Time before vs after; reset the conf after the A/B.

Lever 3 · Use G1GC on large heaps

python# VERSION NOTE: since Spark 4.0.0 (JDK 17 default), G1GC is ALREADY the default collector.
# On older Spark / JDK 8, ParallelGC was default and G1GC was the recommended opt-in:
spark.conf.set(
    "spark.executor.extraJavaOptions",
    "-XX:+UseG1GC -XX:G1HeapRegionSize=16m"    # bump region size for very large heaps
)
# -Xmn tunes the young-gen size when many short-lived objects churn Eden.

# VERIFY — executor stdout shows "[GC pause (G1 Evacuation Pause) ...]" with shorter
# durations; GC Time in the Stages tab falls on large-heap, allocation-heavy stages.

Lever 4 · Off-heap — keep data out of the GC's reach

python# Off-heap memory lives OUTSIDE the JVM heap → the collector never scans it → no GC pressure.
spark.conf.set("spark.memory.offHeap.enabled", "true")
spark.conf.set("spark.memory.offHeap.size", str(2 * 1024 * 1024 * 1024))  # 2 GB, bytes (> 0)
# Shrink the JVM heap accordingly, or you trade GC for OOM (Lesson 04).

# VERIFY — Stages tab: the same cache/shuffle workload shows lower GC Time —
# those bytes left the heap the collector walks.

Figure B · the pause on a timeline

Minor blip vs full stall — and the fix

This figure isolates a different thing from Figure A: not the cluster, but the shape of the pause on a single executor's timeline. Four task-thread bars run in parallel. Trigger a minor GC (a short blip) or a full GC (a long stop-the-world stall) and watch every bar freeze for the pause, then resume together. Then drag spark.memory.fraction down to shrink the Old gen — full-GC pressure eases and the next stall gets shorter. Watch the "full-GC pause" readout fall as you lower the slider.

05 · Decision table

Which lever, and what it costs

Read this by the lever you're about to reach for — it answers "what will this change, and what does it cost?" at a glance. (The why is §3–04; this is the which.)

06 · Edge cases & limitations

Where it gets subtle

New nuance an interviewer probes — not a re-run of the levers:

• A bigger heap can make full GCs worse. More Old-gen memory means each full GC walks more live objects → longer pauses. The fix is usually fewer objects, not more heap.
• Skew amplifies GC. One huge skewed partition (Lesson 08) allocates far more than its peers on one executor, pushing that JVM into GC thrash. Fix the skew first — the GC pressure often disappears with it.
• Spill increases GC. The docs note spilling brings "increased garbage collection" — the spill data structures churn the heap. Curing spill (Lesson 04) often cures the GC symptom too.

Hard limits worth stating plainly:

• You can't eliminate GC. It's intrinsic to the JVM; you only make pauses shorter and rarer. GC tuning is a last-mile lever, not a first resort.
• extraJavaOptions is start-time only. It's read when the executor JVM boots — set it in the cluster's Spark config; you can't flip the collector or GC flags mid-session.
• Off-heap is partly experimental for caching, and you must hand-balance JVM heap vs off-heap size — get it wrong and you trade GC for OOM.
• Version gate. G1GC is the default only since Spark 4.0.0 (JDK 17); on older Spark / JDK 8 it was an opt-in over ParallelGC. State which runtime you mean.

07 · Common mistakes

Gotchas — the mistake and its one-line fix

08 · References

Sources

• Apache Spark — Tuning guide (Memory Management & Garbage Collection Tuning)
• Apache Spark — Configuration (memory.fraction, executor.extraJavaOptions, memory.offHeap.*)
• Apache Spark — RDD Programming Guide (storage levels, serialized caching)
• Azure Databricks — Compute configuration (Spark config / JVM options)