Databricks · One-Pager · Storage Format

Delta Lake log

An open storage layer that wraps Parquet with a file-based transaction log, giving cloud object storage real ACID transactions, time travel, and schema control.

GA Default table format on Databricks Verified June 2026 docs

In one line

Delta Lake is the optimized storage layer for the lakehouse — Parquet data files plus a JSON transaction log (_delta_log) that delivers ACID transactions, versioning, and time travel on plain object storage.

Why it matters

Raw Parquet on S3/ADLS/GCS has no transactions: concurrent writers corrupt data, partial writes leak, and there is no rollback. Delta Lake adds a log on top so a data lake behaves like a database — atomic commits, consistent reads, and "undo" — and it is the default for every table created on Databricks.

Core concepts

ACID on object storage

Atomicity comes from the log: data files are written first, then one commit makes them visible. Uncommitted files from a failed write are simply never referenced.

The transaction log

Each commit is an ordered JSON file in _delta_log; periodic Parquet checkpoints roll up the log so readers don't replay every commit.

Optimistic concurrency

Writers read the latest version, write files, then validate against concurrent commits before committing — no locks; conflicting writes fail instead of corrupting.

Time travel

Every version is a snapshot; query the past with VERSION AS OF / TIMESTAMP AS OF, inspect it via DESCRIBE HISTORY, and roll back with RESTORE.

Schema enforcement & evolution

Writes that don't match the schema are rejected by default; opt into change with mergeSchema or WITH SCHEMA EVOLUTION.

Deletion vectors

Merge-on-read: DELETE/UPDATE/MERGE mark rows as soft-deleted in metadata instead of rewriting whole Parquet files, then apply them at read time.

How a write becomes a readable snapshot

Click a stage

1 · Write files 2 · Commit JSON 3 · Checkpoint 4 · Reader snapshot

Write files — new Parquet data files land in the table directory but are invisible until a commit references them, so a failed write leaves no trace.

Minimal example

SQL · time travel + rollback-- See every version, with timestamp, user, and operation
DESCRIBE HISTORY sales.orders;

-- Read the table exactly as it was at version 42
SELECT * FROM sales.orders VERSION AS OF 42;

-- Or as of a point in time
SELECT * FROM sales.orders TIMESTAMP AS OF '2026-06-01T00:00:00Z';

-- The one line that matters: undo a bad load by restoring a prior version
RESTORE TABLE sales.orders TO VERSION AS OF 41;

Gotchas & limits

VACUUM permanently removes data files past delta.deletedFileRetentionDuration (default 7 days) — time travel to versions referencing those files then breaks.
History (log) is kept for delta.logRetentionDuration (default 30 days), but data-file retention is the real limit on how far back you can travel; raise both together to extend it.
Schema enforcement rejects mismatched writes by default — enable evolution per operation (mergeSchema / WITH SCHEMA EVOLUTION) rather than session-wide to avoid accidental schema drift.
Enabling reader features like deletion vectors upgrades the table protocol and can be irreversible — older clients below the required runtime may no longer read or write the table.

Interview soundbites

Say this out loud

"Delta Lake is Parquet plus a transaction log. The log in _delta_log is what turns a dumb object store into an ACID table — atomic commits, snapshot reads, and time travel."
"Concurrency is optimistic: writers read the current version, write data files, then commit only if no conflicting commit landed first — no locks, conflicting writes fail instead of corrupting."
"Time travel is free because every commit is an immutable snapshot — VERSION AS OF to query the past, RESTORE to roll back, bounded by VACUUM's 7-day file retention."

References

Verified against docs.databricks.com · June 2026