Databricks Asset Bundles (DABs)

The one-liner

The CLI lifecycle — click a step

▸ init → validate → deploy → run → destroy

How it works — deep dive

1 · The databricks.yml skeleton — top-level mappings

Mechanism: one root file defines the bundle. Top-level mappings: bundle, include, variables, artifacts, resources, targets, sync, run_as, workspace, permissions. Only bundle.name is strictly required. Trade-off: a monolith gets unwieldy — split with include.

# databricks.yml — the root file (only bundle.name is strictly required)
bundle:
  name: sales_platform        # logical name; namespaces deployed resources
variables:
  warehouse_id:
    description: SQL warehouse for SQL tasks
    default: ""               # overridable per target / via --var

2 · Structure for scale — include, artifacts, sync

Mechanism: include globs in extra YAML files; artifacts builds things at deploy (e.g. a wheel); sync picks which local paths upload. Trade-off: include paths are relative to root and top-level only — a mis-scoped glob silently includes nothing.

include:
  - resources/*.yml           # jobs.yml, pipelines.yml, dashboards.yml …
artifacts:
  sales_wheel:
    type: whl
    path: ./libs/sales        # built and uploaded on deploy
sync:
  paths: [ ./src ]            # local code synced to the workspace

3 · Resources — declare what gets deployed

Mechanism: under resources, each type (jobs, pipelines, dashboards, alerts, sql_warehouses, models…) holds named declarations mapping to the REST API object. Trade-off: the bundle owns them — UI edits revert on the next deploy.

resources:
  jobs:
    sales_etl:
      name: sales-etl
      tasks:
        - task_key: ingest
          notebook_task: { notebook_path: ./src/ingest.py }
        - task_key: report
          sql_task:
            warehouse_id: ${var.warehouse_id}
            file: { path: ./sql/report.sql }
          depends_on: [{ task_key: ingest }]
  pipelines:
    sales_sdp:
      name: sales-sdp
      libraries: [{ notebook: { path: ./src/pipeline.py } }]

4 · Targets & deployment modes — dev vs prod

Mechanism: targets declares each environment. mode: development namespaces resources per user & pauses schedules (safe to iterate); mode: production uses real names + active schedules. One target is default. Trade-off: wrong target is easy — dev mode renames resources, so check -t.

targets:
  dev:
    mode: development          # prefixes names, pauses schedules — safe to iterate
    default: true
    workspace: { host: https://dev.cloud.databricks.com }
  prod:
    mode: production           # real names, schedules active
    workspace: { host: https://prod.cloud.databricks.com }
    run_as: { service_principal_name: ${var.sp_app_id} }  # who prod runs as

5 · Variables & per-target overrides

Mechanism: variables set defaults; any target overrides them, and --var overrides at the CLI. Reference with ${var.name} (and ${resources....id} to wire resources). Trade-off: empty vars surface only at validate/deploy — set defaults, validate in CI.

# top-level default above; override inside a target:
targets:
  prod:
    variables:
      warehouse_id: 0a1b2c3d4e5f6789   # prod SQL warehouse
# deploy-time override:
# databricks bundle deploy -t prod --var="warehouse_id=..."

6 · The CLI lifecycle

Mechanism: init scaffolds; validate checks (warns on unknown props); summary/plan preview; deploy -t pushes; run executes; destroy -t tears down. Trade-off: destroy is irreversible — guard it in automation.

databricks bundle init                       # scaffold from a template
databricks bundle validate                   # check config (run in CI on PRs)
databricks bundle deploy -t dev              # push to the dev target
databricks bundle run sales_etl -t dev       # execute a deployed job
databricks bundle deploy -t prod             # promote (usually CI on merge)
databricks bundle destroy -t dev             # tear down dev (irreversible)

Targets — one bundle, many environments

▸ click a target to see the deploy command & behavior

Why it matters

• Click-built jobs/pipelines drift between dev & prod and can't be reviewed or rolled back. DABs make deployments reproducible, versioned, automated.
• "Promote a pipeline dev→prod / do CI-CD on Databricks?" → Asset Bundles.

Uses, edge cases & limitations

• Uses: consistent dev→prod promotion, CI/CD, versioning jobs/pipelines/dashboards, team review, importing existing resources via bundle generate + deployment bind.
• Edge cases: secrets/host differ per target — use overrides + secret scopes; development vs production mode changes behavior (dev prefixes names, pauses schedules); run_as sets the prod identity (use a service principal).
• Limitation: DABs deploy Databricks resources, not general cloud infra (use Terraform for that); click-editing deployed resources causes drift — treat the bundle as source of truth.

Common gotchas

• ❌ Click-editing a deployed job → next deploy overwrites it; the bundle is the source of truth.
• ❌ Hard-coding prod host/secrets instead of using targets + variables + secret scopes.
• ❌ Deploying to the wrong target (dev mode renames resources) — check -t.
• ❌ Skipping validate in CI → broken config reaches the workspace.
• ❌ Running destroy against prod by habit — it's irreversible; gate it.