VACUUM, Time Travel
& Retention.

What it is

Analogy Use case

Delta never overwrites data in place. Every change — INSERT, UPDATE, DELETE, MERGE, OPTIMIZE, a deletion-vector op — writes new Parquet files and commits a new version in the transaction log, leaving the old files in storage. Those leftovers power two features and one cost dial:

• Time travel uses the old files — query or restore any past version (VERSION AS OF, TIMESTAMP AS OF, RESTORE).
• VACUUM reclaims the old files — physically deletes unreferenced files older than a retention window to cut storage and purge deleted data.
• Two retention dials govern the trade-off — how long the log/history is kept (delta.logRetentionDuration, default 30 days) and how long the data files are kept (delta.deletedFileRetentionDuration, default 7 days).

Analogy. A Delta table is a document with track changes + version history on. Time travel = "open last Tuesday's version." VACUUM = "permanently purge change history older than N days to shrink the file." The two dials = how long you keep the edit log vs. the actual old text — and you can only reopen a past version if both still exist.

Use case. An analyst's bad DELETE wipes 2 M rows from orders. Because the pre-delete version's files still exist, you RESTORE TABLE orders TO VERSION AS OF 412 in seconds — no backup restore, no downtime. Later, for GDPR, VACUUM makes the deleted rows physically gone.

Why it matters

• Mistakes become reversible. Bad deletes/updates, a broken pipeline run, a wrong overwrite — recover from a past version instead of a backup tape.
• Storage doesn't grow forever. Without VACUUM, every rewrite (OPTIMIZE, MERGE, deletion-vector purge) leaves old files you keep paying to store.
• Compliance needs real deletion. A soft DELETE only hides rows from queries; the bytes survive in old files until VACUUM removes them past retention.
• The trade-off is a dial you own. Longer retention = more recoverability but more storage cost. VACUUM is where you cash in savings — and where you can cut off time travel if too aggressive.

See it work — version timeline & the VACUUM window

A table with versions v0…v8 on a timeline, each with its data files. Drag "now" to age the table, set the VACUUM retention, then click Run VACUUM: versions whose files fall outside the retention window get greyed out — their files are gone, so you can no longer time-travel to them. The earliest live version is the oldest you can still query.

Simplified illustration. Real VACUUM deletes unreferenced files older than deletedFileRetentionDuration (default 7 days); the retentionDurationCheck blocks retention < 7 days unless explicitly disabled. Use DRY RUN to preview, and DESCRIBE HISTORY to find versions.

See it work — the two retention dials

To time-travel N days back you need both the log entry and the data files for that version. So the effective time-travel window is roughly the minimum of the two dials. Drag each slider and watch the effective window — raising only one doesn't extend travel.

Simplified illustration. Effective travel ≈ min(log, data files). DBR 18.0+ requires logRetentionDuration ≥ deletedFileRetentionDuration and blocks travel older than deletedFileRetentionDuration.

How it works — sub-topic by sub-topic

01Every modifying op creates a new version (table history)

Each commit appends a JSON entry to _delta_log and bumps the version by one, leaving old data files in storage until VACUUM removes them. DESCRIBE HISTORY returns versions newest first — columns include version, timestamp, operation, operationMetrics, isolationLevel, isBlindAppend. History is bounded by logRetentionDuration (30 days).

SQL-- Full history, newest first: find the version/timestamp you want.
DESCRIBE HISTORY catalog.schema.orders;

-- Just the latest commit (current version number / last operation).
DESCRIBE HISTORY catalog.schema.orders LIMIT 1;

PySparkfrom delta.tables import DeltaTable
hist = DeltaTable.forName(spark, "catalog.schema.orders").history()   # all versions
hist.select("version", "timestamp", "operation", "operationMetrics").show(truncate=False)

# Last commit version in this session (no extra query):
print(spark.conf.get("spark.databricks.delta.lastCommitVersionInSession"))

02Querying a past version (VERSION AS OF / TIMESTAMP AS OF / @)

Time travel reads the snapshot live at a version/timestamp — it's a read, no table change. Three ways: VERSION AS OF n, TIMESTAMP AS OF '…', and the @ shorthand (t@v123 or t@yyyyMMddHHmmssSSS). You can only reach versions whose files exist (not vacuumed) and whose log entry exists (within log retention).

SQL-- By version number (from DESCRIBE HISTORY)
SELECT * FROM catalog.schema.orders VERSION AS OF 412;

-- By timestamp (resolves to the version live at that instant)
SELECT * FROM catalog.schema.orders TIMESTAMP AS OF '2026-06-20T09:30:00.000Z';

-- @ shorthand: @v<version> or @<yyyyMMddHHmmssSSS> timestamp
SELECT * FROM catalog.schema.orders@v412;
SELECT * FROM catalog.schema.orders@20260620093000000;

PySpark# versionAsOf / timestampAsOf read options
df_v = spark.read.option("versionAsOf", 412).table("catalog.schema.orders")
df_t = spark.read.option("timestampAsOf", "2026-06-20").table("catalog.schema.orders")

03Fixing accidental changes — RESTORE and read-from-past

RESTORE TABLE t TO VERSION/TIMESTAMP AS OF … makes the table's current state equal a past version. It is data-changing (commits a new version re-pointing to old files) and needs MODIFY. For a surgical fix, INSERT/MERGE only the lost rows back from the past version. You cannot restore to a version whose files were vacuumed, and RESTORE is a write so downstream streams may reprocess.

SQL-- Whole-table rollback to a known-good version (data-changing; needs MODIFY)
RESTORE TABLE catalog.schema.orders TO VERSION AS OF 412;
-- ...or to a point in time
RESTORE TABLE catalog.schema.orders TO TIMESTAMP AS OF '2026-06-20T09:30:00.000Z';

-- Surgical: re-insert only the rows a bad DELETE removed, from the past version
INSERT INTO catalog.schema.orders
SELECT * FROM catalog.schema.orders VERSION AS OF 412 AS past
WHERE past.order_id NOT IN (SELECT order_id FROM catalog.schema.orders);

PySparkfrom delta.tables import DeltaTable
dt = DeltaTable.forName(spark, "catalog.schema.orders")
dt.restoreToVersion(412)                          # by version
# dt.restoreToTimestamp("2026-06-20T09:30:00")    # by timestamp

04VACUUM — reclaim unreferenced files older than retention

VACUUM t deletes files that are no longer referenced by the current version and older than the retention threshold (default 7 days, set by delta.deletedFileRetentionDuration). Two reasons: cut storage cost and permanently purge deleted records. Always DRY RUN first — it lists what would be deleted and deletes nothing. VACUUM is the op that actually caps time travel.

SQL-- ALWAYS preview first — lists files that WOULD be deleted, deletes nothing.
VACUUM catalog.schema.orders DRY RUN;

-- Reclaim files older than the table's retention (default 7 days).
VACUUM catalog.schema.orders;

-- Override the window for this run (168 hours = 7 days; shorter is BLOCKED).
VACUUM catalog.schema.orders RETAIN 168 HOURS;

PySparkfrom delta.tables import DeltaTable
dt = DeltaTable.forName(spark, "catalog.schema.orders")
dt.vacuum()        # default retention (7 days)
dt.vacuum(168)     # RETAIN 168 HOURS

05LITE vs FULL VACUUM

• VACUUM t FULL (default): lists every file under the table path — thorough, slower on huge tables; cleans files not in the log (e.g. aborted-txn leftovers).
• VACUUM t LITE (Public Preview, DBR 16.4 LTS+): uses the transaction log to find expired files — much faster on big tables; won't delete files not referenced in the log, and needs a prior successful VACUUM within the 30-day log retention.

SQL-- FULL (default): lists all files; thorough; cleans aborted-txn leftovers.
VACUUM catalog.schema.orders FULL;

-- LITE (Public Preview, DBR 16.4 LTS+): uses the txn log; faster on big tables.
VACUUM catalog.schema.orders LITE;

06The retentionDurationCheck safety check

Databricks blocks VACUUM … RETAIN n HOURS when n is less than 7 days unless you set spark.databricks.delta.retentionDurationCheck.enabled = false. The floor protects in-flight work: a too-short window can delete files a long-running concurrent job still needs, corrupting it or the table. Disabling it is strongly discouraged.

SQL-- This is BLOCKED by default (retention < 7 days):
-- VACUUM catalog.schema.orders RETAIN 1 HOURS;

-- Override ONLY if no long-running job needs those files (discouraged):
SET spark.databricks.delta.retentionDurationCheck.enabled = false;
VACUUM catalog.schema.orders RETAIN 1 HOURS;
SET spark.databricks.delta.retentionDurationCheck.enabled = true;   -- re-enable the guard

07Deletion-vector soft-deletes: REORG … APPLY (PURGE) then VACUUM

With deletion vectors (Lesson 09), DELETE/UPDATE/MERGE mark rows removed without rewriting the Parquet file — the bytes stay. REORG TABLE t APPLY (PURGE) rewrites files to physically apply the soft-deletes; then VACUUM (after retention) removes the now-stale old files. This is how soft-deleted data becomes physically gone for GDPR — a plain DELETE with DVs does not free the bytes.

SQL-- Step 1: physically apply deletion-vector / column-mapping soft-deletes (rewrites files).
REORG TABLE catalog.schema.orders APPLY (PURGE);

-- Faster on large tables: only touch files that actually have soft-deletes.
SET spark.databricks.delta.reorg.purgeMode = rows;
REORG TABLE catalog.schema.orders APPLY (PURGE);

-- Step 2: after the retention window, VACUUM removes the now-stale old files.
VACUUM catalog.schema.orders;

08The two retention dials — and how they bound time travel

delta.logRetentionDuration (default 30 days) keeps the history/log; delta.deletedFileRetentionDuration (default 7 days) is VACUUM's data-file threshold. To time-travel N days back you need both — so effective travel ≈ min of the two. Raising only one doesn't help. DBR 18.0+: logRetentionDuration must be ≥ deletedFileRetentionDuration, and travel older than deletedFileRetentionDuration is blocked.

SQL-- To reliably time-travel 30 days back, raise BOTH dials (data files AND the log).
ALTER TABLE catalog.schema.orders SET TBLPROPERTIES (
  'delta.deletedFileRetentionDuration' = 'interval 30 days',  -- keep DATA FILES 30 days
  'delta.logRetentionDuration'         = 'interval 30 days'   -- keep HISTORY/LOG 30 days (>= data on DBR 18.0+)
);

SHOW TBLPROPERTIES catalog.schema.orders;   -- inspect the current dials

09Time travel to a vacuumed version FAILS (the capstone)

Once VACUUM removes a version's data files, querying or restoring that version (or anything older) errors out — the files it would read are gone. This is the trade-off made concrete: VACUUM saves storage by destroying the very files time travel needs. Order matters — set retention before you rely on a long window.

SQL-- This SUCCEEDS before vacuuming v412's files...
SELECT count(*) FROM catalog.schema.orders VERSION AS OF 412;

-- ...you run VACUUM (short retention, demo only)...
SET spark.databricks.delta.retentionDurationCheck.enabled = false;
VACUUM catalog.schema.orders RETAIN 0 HOURS;
SET spark.databricks.delta.retentionDurationCheck.enabled = true;

-- ...and now the SAME query FAILS: v412's data files were deleted.
-- SELECT count(*) FROM catalog.schema.orders VERSION AS OF 412;   -- ERROR: files no longer exist

10Predictive optimization runs VACUUM for you (Lesson 11)

On UC managed tables, predictive optimization runs VACUUM (and OPTIMIZE/ANALYZE) automatically on serverless — no scheduled job. Because PO's VACUUM uses delta.deletedFileRetentionDuration (default 7 days), raise that property BEFORE enabling PO if you need longer time travel; otherwise PO can delete files you expected to reach.

SQL-- Raise retention BEFORE enabling PO, or PO's VACUUM (7-day default) reclaims files you wanted.
ALTER TABLE catalog.schema.orders
  SET TBLPROPERTIES ('delta.deletedFileRetentionDuration' = 'interval 30 days');
-- Then enable PO at the schema/catalog level (Lesson 11).

Time travel vs VACUUM (and the two dials)

Uses, edge cases & limitations

Uses (when to reach for it)

• Recover from mistakes: RESTORE (whole table) or INSERT/MERGE from a past version (surgical) after a bad DELETE/UPDATE/overwrite.
• Audit & reproducibility: re-run a report against a past day's data; read DESCRIBE HISTORY for who-changed-what.
• Snapshot isolation: pin a stable snapshot of a fast-changing table for a long query/join.
• Reclaim storage & comply: run VACUUM (after retention) to delete stale files and physically purge soft-deleted data.
• When NOT to: as a backup/DR strategy (use real backups), or to reach back weeks on a table where you never raised retention.

Edge cases an interviewer probes

• Vacuumed version → time travel/RESTORE to it fails (files gone).
• Raised only one dial → keep the log but vacuum files = query fails; keep files but prune the log = version not referenceable. Raise both.
• Deletion-vector soft-deletes → a plain DELETE doesn't free bytes; need REORG … APPLY (PURGE) then VACUUM.
• RETAIN < 7 days → blocked by retentionDurationCheck; disabling it can delete files a concurrent job still needs.
• RESTORE & streaming → it's a write, so downstream streams may reprocess.
• DBR 18.0+ → logRetentionDuration must be ≥ deletedFileRetentionDuration; travel older than the data-file window is blocked.
• Big-table VACUUM cost → phase 1 lists files on workers (parallel); phase 2 deletes from the driver (single node) — size the driver up.

Limitations

• VACUUM is irreversible — no undo for deleted files.
• Default retention 7 days (data files) / 30 days (log); longer travel needs raising both properties in advance.
• LITE VACUUM is Public Preview (DBR 16.4 LTS+), needs a prior successful VACUUM within log retention, and won't delete non-log-referenced files.
• Time travel is not a backup — rely only on the past 7 days unless retention was raised.
• VACUUM ignores dirs beginning with _/. (e.g. _delta_log, _checkpoints).

Common gotchas

• DRY RUN first. Preview which files VACUUM would delete before running for real on an unfamiliar table.
• VACUUM kills old time travel. After it runs, versions whose files were removed can't be queried or restored — set retention before you need a long window.
• Raise BOTH dials, not one. Effective travel ≈ min(log, data-file). On DBR 18.0+, logRetentionDuration must be ≥ deletedFileRetentionDuration.
• Soft-delete ≠ purged. With deletion vectors, run REORG … APPLY (PURGE) then VACUUM to physically remove deleted bytes (GDPR).
• Don't disable retentionDurationCheck. Sub-7-day VACUUM can corrupt long-running concurrent jobs.
• RESTORE is a write. Needs MODIFY and may make downstream streams reprocess.
• Not a backup. Time travel is for recent recovery/audit, not disaster recovery.
• PO vacuums for you. Raise delta.deletedFileRetentionDuration before enabling predictive optimization (Lesson 11) if you need long travel.