Databricks · One-Pager · Data Engineering

Lakeflow Declarative Pipelines

Define the tables you want; the framework figures out the dependency graph, the compute, and the incremental updates — so you describe what, not how.

GA SQL & Python · serverless default formerly Delta Live Tables / DLT Verified Jun 2026 docs

In one line

Lakeflow Declarative Pipelines is a declarative framework for building batch and streaming ETL pipelines in SQL or Python, where you define datasets and the engine auto-orchestrates their dependencies, execution, and data-quality enforcement.

Why it matters

Hand-rolled ETL means wiring task order, checkpoints, retries, and quality checks yourself across dozens of jobs. Here you declare each table once; the pipeline derives the DAG, runs stages in the right order with the right parallelism, and tracks streaming state for you — the default path for medallion (bronze → silver → gold) pipelines on Databricks.

Core concepts

Pipeline + flows

A pipeline is the container of datasets; a flow reads a source, applies logic, and writes a target. The pipeline analyzes dependencies and orchestrates order and parallelism.

Streaming table

UC-managed table for append-only sources — each record processed exactly once. Use for ingestion and incremental processing of continuously growing data.

Materialized view

UC-managed batch target whose results are recomputed (incrementally when possible) to reflect current source state. Use for transforms and aggregations when sources change via update/delete.

Expectations

SQL-boolean data-quality constraints checked per record — keep (warn, default), drop the row, or fail the update.

Automatic DAG

Reading one dataset from another declares the edge; the framework infers the full graph — no manual task ordering.

Serverless

Default and recommended compute: Databricks-managed, horizontal + vertical autoscaling, no cluster config.

How it works

Click a stage — the engine builds this DAG from your dataset definitions

1 · Bronze ST 2 · Silver MV + expectations 3 · Gold MV 4 · Auto-orchestration

Bronze streaming table — ingests raw files/events append-only from cloud storage or a message bus; each record processed exactly once.

Minimal example

Python · pyspark.pipelines (was: dlt)from pyspark import pipelines as dp

@dp.table                                   # streaming table (returns a streaming read)
@dp.expect_or_drop("valid_id", "id IS NOT NULL")  # the line that matters: drop bad rows
def bronze():
    return spark.readStream.format("cloudFiles") \
        .option("cloudFiles.format", "json").load("/Volumes/raw/events")

@dp.materialized_view                         # silver MV; dep on bronze inferred automatically
def silver():
    return spark.read.table("bronze").dropDuplicates(["id"])

Gotchas & limits

Default expectation policy is retain — invalid records are written and only counted; you must add ON VIOLATION DROP ROW / FAIL UPDATE to act on them.
Expectation constraints are pure SQL booleans — no custom Python UDFs, external calls, or subqueries against other tables.
Streaming tables are append-only / exactly-once; pick a materialized view when the source mutates via updates and deletes, not a streaming table.
Datasets are pipeline-managed — don't write to a pipeline's tables from outside; FAIL UPDATE aborts the whole update on a single bad row.

Interview soundbites

Say this out loud

"Lakeflow Declarative Pipelines — formerly Delta Live Tables — is declarative ETL: I define streaming tables and materialized views, and the engine builds the dependency DAG and runs it."
"Streaming table = append-only, exactly-once ingestion; materialized view = recomputed (incrementally when possible) to match current source state."
"Expectations are per-record SQL constraints with three policies — keep/warn by default, drop the row, or fail the update — and serverless is the default compute."

References

Verified against docs.databricks.com · June 2026