Field Engineering enablement · Delta Optimization track

Delta Lake
Optimization.

Nine connected lessons on how Databricks physically stores, skips, compacts, and maintains your Delta data — told as one story: a write makes files, layout decides what queries can skip, maintenance keeps files right-sized, and the platform can do all of it for you. Every lesson is a self-contained interactive page with a runnable notebook and a markdown companion.

11topics
11notebooks
22+interactive diagrams
Jun 2026doc-verified

The whole track in one picture

Click each rung to see what it does and what replaced it. The track teaches every tool — so you can maintain legacy tables and design modern ones.

The Delta layout & maintenance ladder

From the raw problem to the fully-managed answer.

Select a rung above.
The one-line modern default For a new table: use a Unity Catalog managed table with liquid clustering on your top filter/join columns and turn on predictive optimization — then let the platform handle layout and maintenance. Reach for partitioning, Z-order, or manual OPTIMIZE only for legacy/external tables or special very-large-table cases.

1 · The baseline

Why naive writes create thousands of tiny files — the problem everything else solves.

01

Traditional writes & the small-file problem

  • One file per write task; per-file overhead
  • coalesce vs repartition; AQE
  • Parquet files + the _delta_log commit
Open lesson → mdnotebook

2 · Classic layout — and why queries can skip

The original layout tool, the statistics that make layout matter, and the Z-order curve.

02

Partitioning (and when NOT to)

  • Don't partition tables < 1 TB; ≥ 1 GB per partition
  • Partition explosion on high-cardinality keys
  • Ingestion-time clustering; convert to liquid
Open lesson → mdnotebook
03

Data skipping & Z-ordering

  • Per-file min/max/null/count stats; first 32 cols
  • dataSkippingStatsColumns, ANALYZE … COMPUTE DELTA STATISTICS
  • ZORDER BY and the Z-order curve
Open lesson → mdnotebook

3 · Compaction & automatic file sizing

The manual fix, then the two ways Databricks gets file sizes right for you, plus the umbrella.

04

OPTIMIZE / compaction (bin-packing)

  • Bin-packing many small files into right-sized ones
  • Idempotent; WHERE; OPTIMIZE FULL (16.0+)
  • Integrates with Z-order & liquid clustering
Open lesson → mdnotebook
05

Optimized writes

  • Shuffle before write → fewer, larger files (128 MB)
  • delta.autoOptimize.optimizeWrite
  • Always on for MERGE / UPDATE / DELETE
Open lesson → mdnotebook
06

Auto compaction

  • Post-write, synchronous on the write cluster
  • delta.autoOptimize.autoCompact = auto/true/false
  • Background auto compaction for UC managed tables
Open lesson → mdnotebook
07

Auto optimize & file-size autotuning

  • optimizeWrite + autoCompact together
  • delta.targetFileSize; autotune 256 MB → 1 GB
  • Not a replacement for OPTIMIZE on big tables
Open lesson → mdnotebook

4 · The modern layout

The layout technique that replaces partitioning + Z-order for new tables.

08

Liquid clustering

  • CLUSTER BY / CLUSTER BY AUTO; GA on DBR 15.4 LTS+
  • Change keys anytime — no rewrite of existing data
  • OPTIMIZE incremental · OPTIMIZE FULL recluster
Open lesson → mdnotebook

5 · Modifications, lifecycle & automation

How edits avoid rewriting files, how to manage the file history they leave behind, and how to hand it all to the platform.

09

Deletion vectors (merge-on-read)

  • DELETE/UPDATE/MERGE without rewriting whole files
  • delta.enableDeletionVectors; soft-deletes
  • REORG … APPLY (PURGE) to apply physically
Open lesson → mdnotebook
10

VACUUM, time travel & retention

  • VERSION AS OF / TIMESTAMP AS OF · RESTORE
  • VACUUM reclaims unreferenced files (7-day default)
  • Two dials: logRetentionDuration + deletedFileRetentionDuration
Open lesson → mdnotebook
11

Predictive optimization

  • Auto OPTIMIZE / VACUUM / ANALYZE on UC managed tables
  • On by default for newer accounts; serverless billing
  • ALTER CATALOG/SCHEMA … ENABLE PREDICTIVE OPTIMIZATION
Open lesson → mdnotebook

Plan & reference

↳

Learning plan

  • Suggested order, timings, and self-check questions
  • The decision framework for new vs legacy tables
Open plan → md