Databricks · One-Pager · Compute

Compute & cluster types

The compute that runs your code — all-purpose, jobs, SQL warehouses, serverless — plus the knobs (access modes, policies, autoscaling, spot, Photon, pools) interviewers love to probe.

GA Photon on by default (DBR 9.1 LTS+) Verified June 2026 docs

In one line

Compute is the set of cluster types that run your workloads — classic (all-purpose, jobs) deployed in your cloud account, or serverless and SQL warehouses managed by Databricks.

Why it matters

Picking the right compute is the single biggest lever on cost and performance: interactive all-purpose clusters are convenient but pricey for automation, while jobs compute and serverless are purpose-built for scheduled work — Databricks does not recommend all-purpose compute for production jobs.

Core concepts

All-purpose vs jobs compute

All-purpose runs interactive, shared notebook work; jobs compute is created per run for a job and torn down after — cheaper and isolated, so it's the choice for scheduled pipelines.

SQL warehouses

Compute optimized for SQL, analytics, and BI; offered classic or serverless, with Photon on by default.

Serverless compute

Databricks-managed, on-demand compute for notebooks, jobs, and Lakeflow Declarative Pipelines — no provisioning, fast start, available by default in most workspaces.

Access modes

Standard (← Shared): multi-user with Lakeguard isolation, Python/SQL/Scala. Dedicated (← Single user): one user or group, adds R. No-isolation is the legacy non-UC mode.

Cluster policies

Admin-defined rules that constrain compute creation — instance types, autoscaling bounds, max DBUs/hour — to control cost and standardize configs.

Autoscaling + spot

Min/max workers scale with load; the driver is always on-demand while workers can be spot to cut cost on interruptible nodes.

Which compute when

Click a compute type

All-purpose Jobs compute SQL warehouse Serverless

All-purpose — use when you're doing interactive, exploratory notebook work shared across a team; not recommended for production jobs.

Minimal example

Jobs compute cluster spec (JSON){
  "new_cluster": {
    "spark_version": "15.4.x-scala2.12",
    "data_security_mode": "SINGLE_USER",   // access mode: dedicated
    "node_type_id": "i3.xlarge",
    "autoscale": { "min_workers": 2, "max_workers": 8 },
    "runtime_engine": "PHOTON",                 // turn on Photon
    "aws_attributes": { "availability": "SPOT_WITH_FALLBACK" },
    "policy_id": "A1B2C3D4E5"                   // enforce a cluster policy
  }
}

Gotchas & limits

All-purpose compute for scheduled jobs is wasteful — it stays up and is billed at the higher interactive rate; use jobs compute or serverless instead.
Access mode is tied to Unity Catalog: Standard and Dedicated support UC; the legacy no-isolation mode does not, so reach for Standard/Dedicated when you need governance.
The driver node is always on-demand; only workers can be spot, so a fully spot cluster still pays on-demand for the driver.
Photon helps long-running, data-heavy queries; sub-two-second queries see little gain since planning/scheduling dominates.

Interview soundbites

Say this out loud

"All-purpose is for interactive notebooks; jobs compute spins up per run and tears down after — cheaper and isolated, so it's what you use for production jobs."
"Access modes were renamed: Standard is the old Shared (multi-user, Lakeguard isolation), Dedicated is the old Single user; both are Unity Catalog-capable, no-isolation isn't."
"Photon is a drop-in vectorized C++ engine — no code change, on by default on serverless, SQL warehouses, and DBR 9.1 LTS and above."

References

Verified against docs.databricks.com · June 2026