Databricks DE · Topic 7.1

Lakeflow Jobs & Tasks

Orchestrate multiple tasks — notebooks, SQL, and pipelines — as one workflow DAG, scheduled and monitored, defined in the UI or as an Asset Bundle. (Formerly Databricks Workflows / Jobs.) This deep dive walks the model and the config-as-code.

job = workflowtask = stepdepends_on DAGAsset Bundle

The one-liner

Lakeflow Jobs coordinates and runs multiple tasks (notebook / SQL / Python / SDP pipeline) as one workflow, wired into a DAG with dependencies, schedules, retries, and monitoring — defined in the UI or as code (Asset Bundle).

🍳 Analogy: a job is a recipe; tasks are the steps. The DAG is the order — "make the sauce before you plate" — and the runner makes each step wait for the ones it depends on.

Run a job — watch the task DAG

▸ click Run; tasks execute in dependency order

Ingest

notebook

▸

Transform

SDP pipeline

▸

DQ check

SQL

▸

Notify

notebook

Idle.

1 · Task types (mix them in one job)

Task type	YAML key	Runs
Notebook	`notebook_task`	a notebook
SDP pipeline	`pipeline_task`	a Lakeflow Declarative Pipeline
SQL	`sql_task`	query / file / dashboard / alert on a warehouse
Python wheel	`python_wheel_task`	an entry point in a packaged wheel
Python file / dbt / Run-job	`spark_python_task` / `dbt_task` / …	scripts, dbt, nested jobs

2 · Config-as-code — a multi-task job (Asset Bundle)

resources.jobs.<key>.tasks[] — each task has a task_key, a task-type block, and depends_on to wire the DAG.

# databricks.yml (resources)
resources:
  jobs:
    nightly_etl:
      name: nightly_etl
      tasks:
        - task_key: ingest                         # task 1
          notebook_task: { notebook_path: ../src/autoloader_ingest.py }
          job_cluster_key: etl_cluster
        - task_key: transform                      # task 2: runs an SDP pipeline
          depends_on: [{ task_key: ingest }]
          pipeline_task: { pipeline_id: ${resources.pipelines.silver_gold.id} }
        - task_key: dq_check                       # task 3: SQL on a warehouse
          depends_on: [{ task_key: transform }]
          sql_task: { warehouse_id: ${var.warehouse_id}, file: { path: ../src/dq_checks.sql } }
        - task_key: notify                         # task 4
          depends_on: [{ task_key: dq_check }]
          notebook_task: { notebook_path: ../src/notify.py }
      job_clusters:                                 # reusable cluster for the job
        - job_cluster_key: etl_cluster
          new_cluster:
            spark_version: 15.4.x-scala2.12
            node_type_id: i3.xlarge
            num_workers: 2

Deploy & run (CLI)

databricks bundle deploy -t prod        # create/update the job
databricks bundle run nightly_etl       # trigger a run

3 · Which cluster runs a task?

Job clusters (job_clusters + job_cluster_key) — created for the run, torn down after; cheapest for scheduled jobs, isolated per run.
Serverless — no cluster to manage; fast start, good default for many tasks.
All-purpose (shared) — avoid for production (cost + noisy neighbors); fine for dev.

4 · Jobs vs SDP — click each

▸ a job can run an SDP pipeline as a task

Lakeflow Jobs

Declarative Pipelines (SDP)

Lakeflow Jobs — orchestrate.

Unit = task (notebook / SQL / Python / pipeline)
You define the DAG of steps + schedule
Runs an SDP pipeline via pipeline_task

SDP — build one pipeline.

Unit = dataset (streaming table / MV)
You declare desired tables; SDP derives its own DAG
Runs inside / alongside Jobs

	Lakeflow Jobs	SDP
Role	Orchestrate tasks	Build a pipeline
Unit	Task	Dataset
DAG	You define it (`depends_on`)	Derived from datasets

Why it matters

Real pipelines are multi-step (ingest → transform → publish → notify) — Jobs run them reliably, scheduled, with dependencies & retries.
"How do you orchestrate a pipeline on Databricks?" → Lakeflow Jobs (and knowing it ≠ SDP).

🌙 Real-world use case: a nightly job runs ingest notebook → SDP pipeline → SQL DQ check → notify, as one DAG, retried on failure, deployed via an Asset Bundle.

Uses, edge cases & limitations

Uses: scheduling end-to-end pipelines; chaining ingest → SDP → checks → notify; mixing notebooks, SQL, ML, pipelines; CI/CD via Asset Bundles.
Edge case: the DAG must be acyclic (no cyclic depends_on); a failed upstream task blocks dependents — configure retries / conditional if-else; libraries can't be declared on a shared job cluster.
Limitation: Jobs orchestrate, they don't replace SDP's incremental engine; one giant task loses per-task retry/observability — split into tasks.

Common gotchas

❌ Confusing Jobs (orchestration) with SDP (pipeline building) — a job can run an SDP pipeline via pipeline_task.
❌ Calling it "Workflows" as current — it's Lakeflow Jobs now.
❌ One monolithic task instead of a task DAG → no granular retries/monitoring.
❌ Creating a dependency cycle — the DAG must be acyclic.
❌ Running production jobs on an all-purpose cluster — use job clusters or serverless.