What a Lakehouse Is

The one-liner

Warehouse vs Lake vs Lakehouse

▸ click a tab to compare

How it works — deep dive

1 · The problem — the two-system split

Mechanism: classically you kept a lake (cheap, open, ML) and a separate warehouse (fast BI, ACID), with ETL copying lake → warehouse. Why it hurts: two copies, an extra ETL hop, staleness, drift, double cost/governance.

# The old two-system pain (what the lakehouse removes):
S3/ADLS lake (raw, ML)  --nightly ETL copy-->  Warehouse (BI)
   ^ one copy of truth                              ^ a second, stale copy
   = extra pipeline, drift, double storage + double governance

2 · The lakehouse idea — one open storage layer

Mechanism: keep data once in cheap object storage in an open table format (Parquet managed by Delta), and run both BI SQL and ML on it — no warehouse copy. Trade-off: you must adopt Delta + UC; plain files = just a lake.

-- One governed table in open format — BI and ML both read THIS, no copy.
CREATE TABLE main.sales.orders (
  order_id BIGINT, region STRING, amount DECIMAL(10,2), order_ts TIMESTAMP
) USING DELTA;                     -- Delta is the default format on Databricks

3 · The enabler — an open table format with a transaction log

Mechanism: Delta wraps Parquet with a transaction log adding ACID, schema enforcement, and version history → lake storage that behaves like a warehouse table. Trade-off: the log enables time travel but old versions cost storage until VACUUM.

-- The transaction log gives reliability + time travel for free:
DESCRIBE HISTORY main.sales.orders;                 -- every versioned commit
SELECT * FROM main.sales.orders VERSION AS OF 3;    -- read as of version 3

4 · Governance & decoupled compute — Unity Catalog

Mechanism: Unity Catalog governs every table/file/model with the 3-level namespace catalog.schema.table + lineage + grants. Compute is decoupled — Spark, Photon, SQL warehouses read the same data, scaled independently. Trade-off: set-up cost, but one model for the whole estate.

-- One governance model over the single copy; engines read the same table.
GRANT SELECT ON TABLE main.sales.orders TO `analysts`;   -- UC access control
-- A SQL warehouse, a Spark job, and an ML notebook all query main.sales.orders.

5 · One copy → BI and ML together

Mechanism: the same governed Delta table serves a BI dashboard (SQL) and a model (DataFrame) — no export, no second system. Trade-off: analytical sweet spot — for high-frequency single-row OLTP use an operational store (e.g. Lakebase).

# BI: SELECT region, sum(amount) FROM main.sales.orders GROUP BY region
# ML reads the SAME table — no copy into a separate system:
df = spark.read.table("main.sales.orders")           # one source of truth
features = df.groupBy("region").sum("amount")         # same data BI reports on

How it works — the stack

▸ click each layer to expand

Why it matters

• The old way needed two systems — a lake for raw/ML data + a separate warehouse for BI → copies, extra ETL, drift, double cost.
• A lakehouse is one copy of data, one governance model, serving BI, data engineering, streaming, and ML/AI together.

Uses, edge cases & limitations

• Uses: unified BI + ML + streaming on one copy; replacing a lake-plus-warehouse stack; the bronze→silver→gold medallion flow (1.3).
• Edge case: very low-latency single-row lookups (OLTP) aren't its sweet spot — that's an operational DB job (e.g. Lakebase), not the analytical lakehouse.
• Limitation: the benefits depend on using Delta + Unity Catalog; plain files in object storage with no Delta/UC is "just a data lake" again.

Common gotchas

• ❌ Calling it "just a data lake with SQL." The differentiators are ACID + governance, not the query engine.
• ❌ Thinking you must pick lake or warehouse — the point is one system.
• ❌ Expecting lakehouse tables to behave like an OLTP database for high-frequency point updates.
• ❌ Forgetting the value comes from Delta + UC — open files alone aren't a lakehouse.