Databricks · One-Pager · Apache Spark

Spark architecture & the lazy execution model

How a Spark job goes from your code to parallel work on a cluster: a driver plans, executors run, and nothing happens until an action fires.

Powers all Databricks compute 1 task per partition Verified June 2026 docs

In one line

Spark is a distributed engine where a driver compiles your operations into a plan and schedules it as tasks across executors, evaluating lazily — only when an action is triggered.

Why it matters

Apache Spark is the technology powering compute clusters and SQL warehouses in Databricks. Understanding driver/executor roles, shuffle boundaries, and lazy evaluation is what lets you reason about why a job is slow, where data moves, and what a query plan is actually doing.

Core concepts

Driver & executors

The driver node maintains the SparkContext, interprets your commands, and runs the Spark master that coordinates executors; worker nodes run the executors that do the distributed work (one executor per worker).

Cluster manager

Allocates the worker VMs and hands resources to the application — on Databricks the cluster and Spark session are configured and managed for you.

Jobs → stages → tasks

Each action launches a job; the scheduler splits it into stages at shuffle boundaries, and each stage runs as tasks — one task per partition.

Partitions

The unit of parallelism: data is divided into partitions and each is processed by a single task, so partition count sets how much runs in parallel.

Transformations vs actions

Every operation is a transformation (reads, joins, aggregations, casts — returns a new DataFrame) or an action (writes, count, display) — only actions execute work.

Lazy eval & Catalyst

No logic runs until an action fires; Databricks then considers all transformations behind that action at once and picks an optimal plan based on the physical layout of the data.

How a job runs

Click a stage

1 · Action fires 2 · Driver plans (DAG) 3 · Stages @ shuffle 4 · Tasks on executors

Action → Job — calling an action (e.g. write, count, display) triggers execution and launches one job.

Minimal example

PySparkdf = spark.read.table("sales")          # transformation — lazy, no work yet
result = (df
  .filter(df.region == "EU")            # narrow: stays within a partition
  .groupBy("category").count())          # wide: shuffle → new stage boundary

result.write.saveAsTable("eu_counts")     # ACTION — fires the job: stages run as tasks

Gotchas & limits

Shuffles (wide transforms: groupBy, join, distinct) move data across the network and start a new stage — the usual bottleneck.
collect() / toPandas() pull all results to the driver; on a large dataset that OOMs the single driver node.
Too many tiny partitions means too many tasks and scheduling overhead; too few means under-used cores and memory pressure (aim ~128–256 MB each).
Because evaluation is lazy, an error in a transformation only surfaces when the action runs — not on the line you wrote it.

Interview soundbites

Say this out loud

"The driver builds the plan and schedules; executors run the tasks. One task processes one partition — that's the unit of parallelism."
"Transformations are lazy and just build a DAG; an action triggers a job. The DAG is cut into stages at shuffle boundaries."
"A shuffle is a wide transformation that redistributes data across partitions over the network — it's the expensive part, so I minimize it."

References

Verified against docs.databricks.com · June 2026