Cache and persist

01 · What it is

Cook it once, keep it in the fridge

By default a DataFrame holds only a recipe — its lineage of transformations — not its rows. Every action (count, write, collect) re-runs that recipe from the source. Caching stores the computed partitions so the next action reads them instead of recomputing. Three verbs are all you need:

• df.cache() — shorthand for df.persist() with the default storage level.
• df.persist(level) — same thing, but you pick the storage level: memory / disk / off-heap, replicated or not.
• df.unpersist() — drop the cached blocks and free the memory. Eager (immediate), unlike cache/persist which are lazy.

analogy An uncached DataFrame is a recipe card: every time you want the dish, you cook it from raw ingredients. Caching is cooking it once and putting it in the fridge — later you just reheat. The catch is fridge space (memory) is finite, and stashing food you'll never eat (a read-once DataFrame) just wastes the shelf.

02 · Why it matters

Recompute is the default — and it's expensive

• Spark recomputes by default. Run three actions on the same DataFrame and Spark executes the full lineage — every read, filter, join, shuffle — three times. If that lineage holds an expensive shuffle or wide join, you pay for it on every action.
• Caching trades memory for compute. Materialize once; later actions read the stored blocks. The classic win is iterative / branching work: ML training loops, repeated exploratory queries, or one intermediate result fed into several writes.
• But caching is not free — and over-caching hurts. Cached blocks pin Storage memory (the R region from Lesson 04), which can evict execution memory and trigger spill or GC pressure. Caching a read-once DataFrame is pure overhead. This is the interview trap: the instinct to "cache everything" makes jobs slower.

use case You clean and join a 500 GB events table once, then run 6 aggregations off it for a dashboard build — cache the cleaned DataFrame so the expensive join runs once, not six times.

03 · How it works — deep dive

Lazy lineage, storage levels, and the first action that pays

A DataFrame's transformations are lazy — they only extend the plan, which runs from the source on each action. persist() inserts a checkpoint: the first action computes and stores the partitions; later actions read the stored copy. That is the entire idea. Everything below — the levels, the defaults, the lazy tax, the cleanup — is explained here once; later sections show it, verify it, and help you decide, but don't re-explain it.

3.1 · The storage levels, and where blocks land

A StorageLevel combines a few independent flags: memory? disk? off-heap?, serialized?, and a replication count. Suffix _2 replicates each block to a second node (_3 exists for disk).

• MEMORY_ONLY — RAM only; any partition that doesn't fit is recomputed on demand. This is the RDD default.
• MEMORY_AND_DISK — RAM first, overflow spills to local disk instead of failing. This is the DataFrame default (PySpark enum MEMORY_AND_DISK_DESER).
• DISK_ONLY — local disk only; zero heap/GC pressure from the cache, at disk-read speed.
• OFF_HEAP — serialized blocks stored outside the JVM heap (no GC on them); experimental, needs spark.memory.offHeap.enabled=true and a positive spark.memory.offHeap.size.
• Scala/Java also expose MEMORY_ONLY_SER / MEMORY_AND_DISK_SER — a serialized byte array per partition (smaller heap, CPU to deserialize).

The same five levels, as a mechanism map — where each block lands and in what shape:

PySpark objects are always Pickle-serialized, so the _SER levels aren't separately exposed there; DISK_ONLY_3 does exist in Python.

3.2 · The PySpark serialization nuance (the interview trap)

interview trap Doc-quoted: "In Python, stored objects will always be serialized with the Pickle library, so it does not matter whether you choose a serialized level." So PySpark does not separately expose the _SER levels — the Python-visible set is MEMORY_ONLY(_2), MEMORY_AND_DISK(_2), DISK_ONLY(_2/_3), and OFF_HEAP. (DISK_ONLY_3 does exist in Python.) Asking for MEMORY_ONLY_SER in PySpark is a no-op concept.

3.3 · Lazy materialization — the first action pays

• Mechanism. persist()/cache() only register the level — nothing is computed or stored. The blocks are written during the first action that materializes the DataFrame; that action is slower (it computes and stores), and every later action reads the cache and is fast.
• Gotcha. If you cache() and never run an action, nothing is cached — the Storage tab stays empty and later actions still recompute. Force it deliberately with a cheap count() or a noop write.

3.4 · unpersist() — eager cleanup, and the storage-vs-GC trade-off

• Mechanism. df.unpersist() immediately removes the cached blocks from memory and disk and clears the level. Unlike cache/persist it is eager. Signature: unpersist(blocking=False).
• Why it matters (Rung 3). Cached blocks live in the Storage sub-region R and stay there until evicted or unpersisted — pinning memory that execution and other caches could use, and adding GC pressure. That's the core trade-off: storage held for reuse vs. heap freed for execution. Always unpersist when done.

Figure A · where the blocks land

Pick a level, watch the blocks settle

Pick a base level, then toggle serialized, replicated (_2), and off-heap. Watch where the cached blocks land — RAM, local disk, or off-heap — the resolved level name, and what happens if a block doesn't fit. This figure isolates one thing: the shape and location of the stored copy. (Where it runs across the cluster is Figure B.)

04 · The code — and how to verify it

Four moves, each with its proof

Every technique below is paired with how to prove it worked — the plan node or Spark-UI signal. Apply, then verify; never assume.

Cache a reused DataFrame and confirm it's used

pythonfrom pyspark.storagelevel import StorageLevel

# expensive: a wide join we don't want to recompute on every downstream action
clean = (events.join(devices, "device_id")        # the costly shuffle/join
                .filter("event_ts > '2026-01-01'"))

clean.persist(StorageLevel.MEMORY_AND_DISK)        # pick the level explicitly
clean.count()                                      # FIRST action: materialize the cache (slow once)

# ...now several downstream actions reuse the stored blocks instead of re-joining:
a = clean.groupBy("country").count()
b = clean.where("is_fraud").count()

# VERIFY — a.explain(mode="formatted") reads from the cache, not the join:
#   +- InMemoryTableScan [country#..]            <-- reading the CACHE
#         +- InMemoryRelation [..], StorageLevel(disk, memory, deserialized, 1 replicas)
#   (no SortMergeJoin / no FileScan events here  <-- the join is NOT re-run)

Spark UI → Storage tab: one cached entry, level Disk Memory Deserialized 1x Replicated, Fraction Cached 100%. If it's < 100%, the cache didn't all fit.

Choose a non-default level (too big for memory)

python# The cleaned DataFrame is 2× cluster RAM — don't fight for heap; cache to disk only.
huge.persist(StorageLevel.DISK_ONLY)   # PySpark also exposes DISK_ONLY_2 / DISK_ONLY_3
huge.count()                           # materialize

# VERIFY — Storage tab: Size in Memory = 0, Size on Disk > 0, level "Disk Serialized 1x".

Always unpersist when done

pythonclean.unpersist()        # eager: releases Storage memory for execution / other caches

# VERIFY — the Storage tab entry disappears; later .explain() falls back to the source plan.

Contrast: read-once (don't cache) vs reused (do cache)

python# ANTI-PATTERN: caching a DataFrame used by exactly ONE action — pure overhead.
once = big.filter("active").cache()
once.write.format("noop").mode("overwrite").save()   # only action → cache cost for nothing

# RIGHT: cache only when there are ≥ 2 actions, and unpersist after the last one.
reused = big.filter("active")
reused.persist(StorageLevel.MEMORY_AND_DISK); reused.count()       # materialize once
m1 = reused.groupBy("region").count(); m2 = reused.where("vip").count()  # 2 reuses → pays off
reused.unpersist()

# VERIFY — Spark UI SQL tab: the reused query shows a cached scan node;
# the source stages disappear on the second action.

Figure B · materialize once, reuse many

See it run across the cluster

This figure isolates a different thing from Figure A: not the shape of a block, but what happens across the nodes on each action. Step persist() then two actions across 1 driver + 3 executors, each holding its own block manager. The first action computes and stores the cached blocks; with a replicated level a copy is placed on a peer; the second action reuses them — no recompute. Use Prev / Play / Next.

05 · Decision table

Which level, and what it costs you

Read this by the situation you're in — it answers "which level, and what's the failure mode?" at a glance. (The why is §3; this is the which. The _SER row is Scala/Java only — PySpark always serializes.)

06 · Edge cases & limitations

Where it gets subtle

New nuance an interviewer probes — not a re-run of §3:

• Cache that doesn't fit behaves differently by level. MEMORY_ONLY silently recomputes the missing partitions (you think it's cached, but it isn't); MEMORY_AND_DISK spills them. Always read Fraction Cached — anything < 100% is the tell.
• Over-caching evicts execution memory. Cached blocks pin Storage (R); too many force shuffles/aggregations to spill or push the executor toward OOM (Lesson 04). Cache less, use DISK_ONLY, or just recompute.
• The cache is a snapshot. Mutating the source after caching won't show new rows until you unpersist() and recompute — on Delta, prefer re-reading the table.
• CACHE TABLE is eager, df.cache() is lazy. The SQL form materializes immediately unless you write CACHE LAZY TABLE — easy to mix up.

Hard limits worth stating plainly:

• There's no "cache now." Cache/persist are lazy; only unpersist() is eager. Nothing is stored without an action.
• A level can't be reassigned in place. persist() refuses to change the level of an already-persisted DataFrame — unpersist() first.
• Cache is ephemeral. It's in-cluster and snapshot-in-time — it does not survive a cluster restart.
• Caching is not a substitute for pruning (Lesson 07) or fixing a bad join (Lesson 02) — it reuses an expensive result; it doesn't make that result cheaper to compute.

07 · Common mistakes

Gotchas — the mistake and its one-line fix

08 · References

Sources

• Apache Spark — RDD Programming Guide (persistence, storage levels, the PySpark "always serialized" nuance)
• Apache Spark — SQL Performance Tuning (caching data in memory, CACHE TABLE)
• Apache Spark — PySpark DataFrame.persist / cache / unpersist API
• Apache Spark — pyspark.StorageLevel (the Python-exposed levels)
• Apache Spark — Tuning (memory management, serialized caching & GC trade-off)
• Azure Databricks — Optimize performance with caching (Spark cache vs disk cache)