Databricks · One-Pager · Ingestion

Auto Loader incremental

A Structured Streaming source that ingests new files from cloud storage as they land — no manual bookkeeping of what's already been processed.

GA format("cloudFiles") Exactly-once via checkpoint Verified June 2026 docs

In one line

Auto Loader is the cloudFiles Structured Streaming source that incrementally and efficiently processes new data files as they arrive in cloud object storage, with no additional setup.

Why it matters

It's the default way to do file-based ingestion on Databricks. Auto Loader tracks which files it has already seen in a checkpoint, so you never re-scan or re-process a billion-file landing zone by hand — you just point a stream at a path and it picks up only the new files, exactly once.

Core concepts

cloudFiles source

A streaming reader over a cloud path; supports JSON, CSV, XML, Parquet, Avro, ORC, text, and binaryFile.

Directory listing vs file notification

Listing scans the path (default, zero setup); notification (useNotifications=true) uses a cloud queue — fewer API calls, scales to high file counts.

Schema inference & evolution

Infers from the first 50 GB or 1000 files; schemaLocation persists it. Evolution defaults to addNewColumns (no schema) or none (explicit schema).

Checkpoint / exactly-once

Processed-file metadata is persisted in the checkpoint (RocksDB), guaranteeing each file is processed exactly once across restarts.

Rescued data column

_rescued_data captures fields that don't match the schema — extra columns, type and case mismatches — as JSON, so nothing is silently dropped.

How it works

Click a stage

1 · New files land 2 · Discover 3 · Infer / evolve schema 4 · Write to Delta 5 · Checkpoint

Files land — new objects arrive in S3 / ADLS / GCS / a Unity Catalog volume; you never list them yourself.

Minimal example

PySpark(spark.readStream
   .format("cloudFiles")
   .option("cloudFiles.format", "json")        # source file format
   .option("cloudFiles.schemaLocation", "/Volumes/main/raw/_schema")
   .load("/Volumes/main/raw/landing/")
 .writeStream
   .option("checkpointLocation", "/Volumes/main/raw/_ckpt")  # tracks processed files → exactly-once
   .trigger(availableNow=True)
   .toTable("main.bronze.events"))

Gotchas & limits

cloudFiles.schemaLocation is required for schema inference, and each stream needs its own dedicated checkpoint — sharing one corrupts file tracking.
Directory listing is the default but gets slow on huge buckets; switch to file notification (cloudFiles.useNotifications=true), or file events on DBR 14.3 LTS+, for scale.
With an explicit schema, evolution defaults to none — new columns are dropped unless you set rescuedDataColumn or change the mode.
In addNewColumns mode the stream fails with UnknownFieldException on a new column by design; it adds the column and you simply restart.

Interview soundbites

Say this out loud

"Auto Loader is the cloudFiles streaming source — it tracks processed files in the checkpoint, so it ingests only new files, exactly once, with no manual state."
"Directory listing is the zero-setup default; for high file volumes I flip to file notification mode so it reads a cloud queue instead of re-listing the path."
"Schema drift is handled gracefully — it evolves the schema (addNewColumns) and routes anything that doesn't fit into the _rescued_data column instead of failing the load."

References

Verified against docs.databricks.com · June 2026