Databricks · One-Pager · Streaming

Structured Streaming

A near real-time engine that runs the same Spark DataFrame code over an unbounded stream, giving you exactly-once results via checkpoints.

GA Micro-batch · exactly-once Verified Jun 2026 docs

In one line

Structured Streaming is a near real-time processing engine that runs a query incrementally over an unbounded table, offering end-to-end fault tolerance with exactly-once guarantees using familiar Spark APIs.

Why it matters

You write a streaming job exactly like a batch DataFrame query — Spark turns it into an incrementally updated result and handles fault tolerance, state, and exactly-once delivery for you. It is the engine under Auto Loader and Lakeflow Declarative Pipelines (← Delta Live Tables) and the default for incremental ingestion on Databricks.

Core concepts

Micro-batch model

The stream is processed as a series of small batch jobs; the default trigger runs as fast as possible (~equivalent to processingTime='0 seconds'). Continuous processing is experimental and not recommended.

Sources & sinks

Read with readStream (Delta, Auto Loader, Kafka, Kinesis, rate); write with writeStream to Delta, files, Kafka, or arbitrary sinks via foreachBatch.

Checkpoint = exactly-once

The checkpoint directory stores offsets, commits, state, and the query ID; with a replayable source and idempotent sink this delivers exactly-once. Each query needs its own location.

Output modes

append (only new rows), update (changed rows), complete (full result table each batch). The legal mode depends on the query's aggregations.

Triggers

processingTime for fixed-interval micro-batches; availableNow=True for incremental batch then stop (replaces deprecated Trigger.Once on DBR 11.3 LTS+).

Event-time & watermarks

withWatermark sets an event-time column plus a lateness threshold so Spark can bound state and drop old state for windowed aggregations and stream-stream joins.

How it works

Click a stage

1 · Source 2 · Micro-batch engine 3 · Stateful op + watermark 4 · Sink

Source — a replayable source (Delta, Auto Loader, Kafka) exposes offsets so each micro-batch can read only new data.

Minimal example

PySparkstream = (spark.readStream.format("delta").table("main.bronze.events")
  .withWatermark("event_time", "10 minutes")            # bound state for late data
  .groupBy(window("event_time", "5 minutes"), "id").count())

(stream.writeStream.outputMode("append")
  .option("checkpointLocation", "/Volumes/main/bronze/_ckpt/agg") # exactly-once recovery; unique per query
  .trigger(availableNow=True)                              # incremental batch, then stop
  .toTable("main.silver.events_5min"))

Gotchas & limits

Every query needs its own checkpointLocation — never share one across queries, and deleting it forces a full reprocess.
foreachBatch gives at-least-once, not exactly-once — make the batch function idempotent (e.g. MERGE) yourself.
complete mode keeps the full result in state every batch — it ignores the watermark and does not scale to large key spaces.
Continuous processing is experimental/unsupported; use the default micro-batch trigger (or real-time mode) for low latency.

Interview soundbites

Say this out loud

"Structured Streaming runs your batch DataFrame code as a series of micro-batches over an unbounded table — same API, incremental results."
"Exactly-once comes from the checkpoint: it tracks source offsets and committed batches, so a replayable source plus an idempotent sink means no duplicates and no loss on restart."
"Watermarks (withWatermark) tell Spark how long to wait for late data so it can bound and evict state in windowed aggregations and stream-stream joins."

References

Verified against docs.databricks.com · June 2026