Databricks DE · Topic 3.3

VACUUM & Time-Travel UNDO

VACUUM permanently deletes old, unreferenced data files past the retention window (default 7 days) — reclaiming storage, but ending time travel beyond that point. This deep dive walks the lifecycle and the code.

7-day defaultRETAIN n HOURSDRY RUNLITE / FULLsafety check

The one-liner

VACUUM removes data files no longer referenced and older than the retention threshold (default 7 days). It reclaims storage and finalizes deletes — but versions older than retention are no longer time-travelable afterward.

🗑️ Analogy: VACUUM is emptying the recycle bin. A DELETE just moves old files to "trash" (still time-travelable); VACUUM empties the bin for anything older than retention — space back, but no more undo for those.

Retention vs time travel — drag the threshold

▸ files older than retention get purged on VACUUM

Retention (days): 7

1 · The delete → vacuum lifecycle

Row removal happens in two stages — the model interviewers want: a logical tombstone (files stay, still time-travelable, no space freed), then a physical removal by VACUUM once past retention.

DELETE FROM main.sales.events WHERE event_date < '2026-01-01';  -- stage 1: logical
VACUUM main.sales.events;                                       -- stage 2: physical (past retention)

2 · DELETE vs VACUUM — click each

VACUUM main.sales.events RETAIN 168 HOURS;   -- explicit 7-day retention
VACUUM main.sales.events DRY RUN;            -- preview what would be purged
VACUUM main.sales.events LITE;               -- log-only scan (Public Preview, DBR 16.4 LTS+)

1 · DELETE / UPDATE / MERGE / OPTIMIZE▸

Marks old files "unused" (tombstones) in the log but leaves them on disk. Data is still recoverable via time travel — nothing is freed yet.

2 · VACUUM … DRY RUN▸

Previews exactly which files would be deleted, without removing anything. Always do this first on important tables.

3 · VACUUM▸

Permanently deletes unused files older than retention. Storage reclaimed; time travel to those versions is now gone. Below-threshold VACUUM is blocked by a safety check.

PySpark — DeltaTable API

from delta.tables import DeltaTable
dt = DeltaTable.forName(spark, "main.sales.events")
dt.vacuum()        # default retention
dt.vacuum(168)     # retain 168 hours

3 · The retention safety check (interview trap)

Files are eligible only past delta.deletedFileRetentionDuration (default 7 days). Databricks blocks a below-threshold VACUUM — a long-running query started before the VACUUM may still need files you'd delete.

⚠️ Only if you understand the risk. This can break time travel AND corrupt concurrent readers/writers. Keep retention ≥ your longest job runtime.

SET spark.databricks.delta.retentionDurationCheck.enabled = false;
VACUUM main.sales.events RETAIN 0 HOURS;   -- dangerous; rarely correct

Tune retention (longer rollback window, more storage)

ALTER TABLE main.sales.events
  SET TBLPROPERTIES ('delta.deletedFileRetentionDuration' = 'interval 30 days');

4 · Automation & the compliance angle

Predictive Optimization runs VACUUM automatically on UC managed tables — you often don't schedule it. It does not run automatically otherwise (external tables → you schedule it).
GDPR / right-to-be-forgotten: a DELETE alone isn't enough — data persists in old files until VACUUM physically removes it past retention. Pattern: DELETE the records, then ensure VACUUM runs.

Why it matters

Without VACUUM, every rewrite (OPTIMIZE, MERGE, DELETE) leaves stale files → storage grows forever and listing slows.
"Why can't I VACUUM with 0 retention?" is a classic interview Q — it would break time travel and can corrupt in-flight concurrent readers/writers.

💽 Real-world use case: a heavily-updated silver table balloons in storage. A scheduled VACUUM (7-day retention) clears stale files nightly — keeping a week of rollback while controlling cost.

Logical vs physical removal

	DELETE / MERGE / OPTIMIZE	VACUUM
Effect	tombstones old files (logical)	physically deletes them
Space freed	❌ not yet	✅ (files past retention)
Time travel to old version	✅ still works	❌ gone after VACUUM
Automatic?	n/a	only via Predictive Optimization (managed)

Uses, edge cases & limitations

Uses: reclaim storage, finalize compliance deletions, routine maintenance of churny tables.
Edge case: VACUUM shrinks the time-travel window; LITE mode (Public Preview, DBR 16.4 LTS+) is faster but needs a prior successful VACUUM within log retention. Always DRY RUN first.
Limitation: only removes files past retention — it won't instantly free space from a 5-minute-old delete; lowering retention to force it trades away time travel and risks concurrent readers.

Common gotchas

❌ Setting retention to 0 to "save space now" — breaks time travel & can corrupt concurrent reads; blocked by default.
❌ Expecting time travel to reach old versions after VACUUM — it can't.
❌ Surprised storage didn't drop — only files older than retention are removed.
❌ Assuming DELETE alone satisfies GDPR — you must VACUUM to physically remove.
❌ Manually scheduling VACUUM on managed tables Predictive Optimization already handles.