Databricks DE · Topic 5.1

Declarative Pipelines (SDP)

Describe the result you want in SQL/Python; Lakeflow Spark Declarative Pipelines builds the dependency graph and runs it — batch or streaming. (Formerly Delta Live Tables / DLT.) This deep dive walks the model and the code.

was DLT → SDPstreaming table vs MVpyspark.pipelinesDAG auto-derived

The one-liner

Lakeflow Spark Declarative Pipelines (SDP) — you define datasets (streaming tables, materialized views) + flows; Databricks derives the DAG, order, and incremental processing. It's the current name for Delta Live Tables (DLT).

🧭 Analogy: declarative = a GPS destination, not turn-by-turn directions. You say "clean silver orders from this bronze source"; SDP plans the route (what to run, in what order, incrementally) instead of you hand-coding every step and schedule.

Imperative job vs declarative pipeline

▸ toggle the two styles

Imperative (manual job)

Declarative (SDP)

You write the orchestration yourself:

Sequence tasks & handle retries
Write incremental logic + manage checkpoints
More code, more to break

You declare datasets; SDP handles the rest:

Derives the dependency DAG & run order
Streaming tables handle incremental processing
Framework manages checkpoints & optimization

	Imperative job	Declarative (SDP)
Task ordering	You sequence it	SDP derives the DAG
Incremental logic	You write it	Streaming tables
Checkpoints	You manage	Managed

1 · Streaming table vs materialized view

	Streaming table (ST)	Materialized view (MV)
Reads	a streaming source (`readStream`)	a batch query (`read`)
Semantics	append/incremental; each row once	result kept equal to the query
Use for	bronze/silver ingestion & incremental transforms	gold aggregates / dimensions

💡 Rule of thumb: ingesting/incrementally transforming new rows → streaming table; an aggregate/joined result that must always equal its query → materialized view.

2 · Author in Python (current API)

The Python API now lives in pyspark.pipelines. import dlt still works but pipelines is recommended.

from pyspark import pipelines as dp     # current API (replaces `import dlt`)

@dp.table                                # streaming table — reads a stream
def bronze_orders():
    return (spark.readStream.format("cloudFiles")
              .option("cloudFiles.format", "json")
              .load("/Volumes/main/raw/orders"))

@dp.table
def silver_orders():
    return spark.readStream.table("bronze_orders").dropDuplicates(["order_id"])

@dp.materialized_view                    # batch MV — reads a snapshot
def gold_daily_revenue():
    return spark.read.table("silver_orders").groupBy("order_date").sum("amount")

3 · Author in SQL

-- streaming table: incremental ingest (STREAM = streaming read)
CREATE OR REFRESH STREAMING TABLE bronze_orders
AS SELECT * FROM STREAM read_files('/Volumes/main/raw/orders', format => 'json');

CREATE OR REFRESH STREAMING TABLE silver_orders
AS SELECT * FROM STREAM bronze_orders;          -- depends on bronze → SDP orders it

-- materialized view: a gold aggregate kept in sync with its query
CREATE OR REFRESH MATERIALIZED VIEW gold_daily_revenue
AS SELECT order_date, sum(amount) AS revenue FROM silver_orders GROUP BY order_date;

4 · You declare datasets → SDP builds the graph

▸ click each dataset

bronze_orders

streaming table

▸

silver_orders

streaming table

▸

gold_daily

materialized view

bronze_orders — a streaming table ingesting raw orders (e.g. Auto Loader). You just declare it; SDP runs it incrementally.

silver_orders — a streaming table that cleans/dedupes bronze. SDP sees it depends on bronze and orders the run automatically.

gold_daily — a materialized view aggregating silver. SDP refreshes it from silver — you never wrote the schedule or the DAG.

A normal update processes only new data; a full refresh recomputes from scratch (re-reads all source data, resets ST state). Development mode reuses the cluster & skips retries for fast iteration; production mode restarts/retries on failure.

SDP vs Lakeflow Jobs

	Declarative Pipelines (SDP)	Lakeflow Jobs (Topic 7)
Purpose	build an ETL pipeline (datasets + flows)	orchestrate tasks
Unit	streaming tables / materialized views	tasks (notebook, SQL, pipeline…)
Ordering	derived from data dependencies	you wire task dependencies
Together	a Job can run an SDP pipeline as one task, alongside notebooks + SQL

Why it matters

Hand-built pipelines mean you write the orchestration (order, retries, incremental, checkpoints). SDP removes that boilerplate.
"DLT vs Lakeflow Declarative Pipelines?" and "imperative vs declarative?" are common interview questions — know the rename and the model.

🥉🥈🥇 Real-world use case: a medallion pipeline. You declare each table + its query; SDP builds the dependency graph, runs bronze→silver→gold in order, and incrementally updates only what changed.

Uses, edge cases & limitations

Uses: medallion ETL, incremental batch + streaming transforms, pipelines with data-quality rules & SCD (5.2–5.4).
Edge case: declarative ≠ magic — a wrong dependency or non-incremental query still costs; full refresh vs incremental update behave differently (full refresh re-reads all source data & resets streaming state). Streaming-table sources should be append-only.
Limitation: it's a framework with its own pipeline runtime/modes — for plain orchestration of notebooks/SQL use Lakeflow Jobs (Topic 7). Docs/UI may still say "DLT" in places.

Common gotchas

❌ Treating "DLT" as different — DLT = Lakeflow Spark Declarative Pipelines now.
❌ Using import dlt in new code — prefer from pyspark import pipelines (old import still runs).
❌ Expecting it to replace Lakeflow Jobs — SDP builds the pipeline; Jobs orchestrate pipelines + notebooks + SQL.
❌ Materialized view where you need append/incremental (use a streaming table), or vice-versa.
❌ Writing imperative orchestration inside a declarative pipeline — let SDP derive order from dataset definitions.