Databricks Architecture at a Glance

The one-liner

How it works — the two planes

▸ click each block to expand

How it works — deep dive

1 · Control plane vs compute plane — what lives where

Mechanism: the control plane (Databricks' account) hosts UI, scheduler, notebooks, APIs; the compute plane runs the clusters/warehouses. Control plane issues work, compute plane does it. Trade-off: control plane is always Databricks-managed — you choose only the compute variant.

# A CLI/API call hits the CONTROL plane (scheduler/APIs); the job runs on the
# COMPUTE plane. Same split whether you click the UI or call the API.
databricks jobs run-now --job-id 620 --profile prod

2 · Compute-plane variants — serverless vs classic

Mechanism: serverless runs in a serverless compute plane in Databricks' account (seconds to start, near-zero ops); classic runs in your account/VPC (natural isolation, you size it). Trade-off: isolation/control vs speed/low-ops.

# Serverless: a SQL warehouse with serverless enabled (no cluster to size).
resources:
  sql_warehouses:
    bi_wh: { name: bi, cluster_size: Small, enable_serverless_compute: true }
  # Classic: a job cluster you size, running in your account's compute plane.
  jobs:
    etl:
      job_clusters:
        - job_cluster_key: main
          new_cluster: { spark_version: 15.4.x-scala2.12, node_type_id: i3.xlarge, num_workers: 4 }

3 · Account → workspaces → metastore (Unity Catalog)

Mechanism: one account holds many workspaces; UC is account-level, with one metastore per region that region's workspaces attach to. Trade-off: workspaces must match the metastore's region; isolate data at the catalog level.

-- UC governs every workspace via the 3-level namespace:
-- (metastore → catalog → schema → table)
SELECT * FROM prod_catalog.sales.orders;   -- catalog.schema.table
-- One regional metastore backs all workspaces in that region; isolate by catalog.

4 · Compute types — match the tool to the job

Mechanism: all-purpose clusters (interactive dev), jobs clusters (ephemeral, per-run), SQL warehouses (BI/SQL), and serverless variants — all read the same UC data. Trade-off: wrong tool wastes money (always-on cluster for scheduled ETL; a cluster for BI).

# Right tool per workload (all read the same governed tables):
#   all-purpose cluster → interactive dev    (attach a notebook)
#   jobs cluster        → scheduled ETL      (ephemeral job_clusters)
#   SQL warehouse       → dashboards / BI SQL (resources.sql_warehouses)
#   serverless          → instant, low-ops    (enable_serverless_compute: true)

5 · How a query/job flows through the planes

Mechanism: submit via the control plane → it uses compute-plane clusters/warehouses → those read/write your data in cloud storage (UC-governed) → results return to the UI. Trade-off: more moving parts than a single box, but each scales independently.

You ──(control plane: UI/API/schedule)──▶ provision compute
        compute plane (classic VPC or serverless) ──reads/writes──▶ your cloud storage
        UC governs access at every step ──▶ results back to the control-plane UI

Classic vs Serverless compute

▸ click a tab

Uses, edge cases & limitations

• Uses: picking the right compute model; explaining where data lives in a security review; reasoning about cost (idle clusters vs pay-per-use).
• Edge case: strict data-residency / no-egress requirements push to classic in your own VPC; serverless availability varies by region/feature.
• Limitation: the control plane is always Databricks-managed — not self-hostable. Don't delete workspace storage: it's tied to control-plane state and is unrecoverable.

Common gotchas

• ❌ Thinking Databricks "stores your data." It orchestrates; your tables live in your cloud storage (or serverless managed storage), governed by UC.
• ❌ Confusing workspace storage (notebooks/logs) with data storage (UC tables/Volumes).
• ❌ Assuming serverless = classic with less setup — the compute runs in a different account with different isolation & networking.
• ❌ Expecting a workspace to attach to a metastore in another region — it must match the regional metastore.