Databricks · One-Pager · Orchestration

Lakeflow Jobs orchestration

The native scheduler for the lakehouse: wire tasks into a DAG, trigger them on a clock or on new data, and run them on cheap ephemeral compute.

GA formerly Workflows Serverless default where supported Verified Jun 2026 docs

In one line

Lakeflow Jobs (formerly Workflows) is Databricks' built-in orchestrator that schedules and runs a workflow of dependent tasks as a directed acyclic graph (DAG).

Why it matters

It's the native, no-extra-infra way to chain notebooks, pipelines, SQL, and dbt into a production data flow with dependencies, retries, alerting, and identity — so you orchestrate inside the platform instead of standing up a separate Airflow.

Core concepts

Jobs & tasks

A job orchestrates tasks; a task is "a specific unit of work within a job." One job spans a single task to hundreds.

Task DAG (Depends on)

Tasks form a DAG: each task's Depends on sets its upstream tasks, giving sequential and fan-out/fan-in execution.

Task types

Notebook, Python script/wheel, SQL, Pipeline (Lakeflow Declarative Pipelines), dbt, JAR, Spark Submit, Run Job, plus If/else and For each control flow.

Triggers & schedules

Scheduled (cron/Quartz), file arrival on a Unity Catalog location, continuous, and table-update triggers — or run manually.

Compute: job vs all-purpose

Serverless is the default where supported; job clusters spin up per run and terminate after. All-purpose works but is "not recommend[ed]" for production.

Retries, alerts & Run as

Per-task retries with interval and timeout; notifications on start/success/failure; the Run as identity (user or service principal) governs all data and compute access.

How it works

Click a task in the DAG

1 · Ingest 2 · Transform 3 · DLT Pipeline 4 · Notify

Ingest — a notebook or Python task lands raw files into a bronze table; a file-arrival trigger can fire the whole job when new data appears.

Minimal example

Databricks Asset Bundle · databricks.ymlresources:
  jobs:
    daily_etl:
      name: daily_etl
      trigger: { periodic: { interval: 1, unit: DAYS } }
      tasks:
        - task_key: ingest
          notebook_task: { notebook_path: ./ingest.py }
        - task_key: transform
          depends_on: [{ task_key: ingest }]   # DAG edge: transform waits for ingest
          notebook_task: { notebook_path: ./transform.py }
          max_retries: 2

Gotchas & limits

All-purpose compute runs jobs but is "not recommend[ed]" for production — reusing an interactive cluster costs more and couples jobs to it; prefer serverless or job clusters.
File-arrival triggers check "every minute" (best effort) and, without file events, cap at 50 trigger-configured jobs per workspace and 10,000 files in the watched location. Re-uploading the same filename does not fire.
The scheduler "is not intended for low-latency jobs" — runs can be delayed several minutes, and the minimum schedule interval is 10 seconds.
By default a job runs as its creator; Databricks recommends setting Run as to a service principal so privileges are decoupled from a person.

Interview soundbites

Say this out loud

"Lakeflow Jobs — formerly Workflows — is the lakehouse-native orchestrator: tasks in a DAG with Depends on edges, triggered by schedule, file arrival, or continuously."
"Use job clusters or serverless, not an all-purpose cluster, for scheduled production jobs — ephemeral compute is cheaper and isolates the run."
"The Run as identity, not the trigger-er, decides what data and compute the job can touch — point it at a service principal in production."

References

Verified against docs.databricks.com · June 2026