The Two Planes and Three Doors, Explained Simply

Control Plane vs Compute Plane: Who Owns What

• Control plane (the brain) - the dispatch office that directs work; Databricks runs it.
• Compute plane (the muscle) - the trucks that process your data; this is where Spark actually runs.
• Subscription - an Azure account that owns and pays for resources; "whose subscription" just means whose bill and whose control.

Two separate sets of machines, run by two parties, in two subscriptions.

• Control plane = the brains. Web UI, REST APIs, job/cluster managers, Unity Catalog metadata and access decisions, and the SCC relay. Databricks owns and runs it. You get no network knobs inside it.
• Compute plane = the muscle. Where clusters and SQL warehouses actually run Spark. The only real difference between flavors is whose subscription the VMs live in.

Interactive: click the boxes

Click each box to see who owns it and what runs there.

How the brain runs the muscle (without dialing in)

If the two planes are separate machines in separate subscriptions, how does the control plane "run this" on compute it can't connect into? Through a reverse tunnel via the SCC relay (which lives in the control plane):

1. The compute node dials out first. At launch each No-Public-IP cluster VM opens one outbound 443 connection to the SCC relay - never the other way around (this is Door ②).
2. The connection is held open. The control plane then pushes run / scale / terminate commands back down that already-open tunnel.
3. The direction of connection setup (outbound, from compute) is not the direction of commands (control plane → compute). That is how "the brain directs the muscle" with no inbound port on your network.

Why the plane decides which toolbox is on the table

In classic, compute is in your VNet, so you can attach an NSG, a UDR, a firewall, or a Private Endpoint. In serverless, there is no VNet of yours, so none of those apply. That is exactly why NCC (Network Connectivity Configuration) exists - the serverless answer to "VNet controls" (Stage 5).

The Secure Boundary: Connections Point Outward

• No inbound ports - nothing on the internet can open a connection into your compute (no doorbell).
• Outbound / reverse connection - the cluster dials out first, and replies come back on that same open line.
• SCC relay - the "switchboard" in the control plane that the cluster calls, so the brain can send commands down the line it answered.

The platform's single most important security property is "no inbound ports." It is not magic - it is a reversed connection.

How it works with SCC / No Public IP (the default)

1. Cluster VMs have no public IPs; your VNet has no open inbound ports. Nothing on the internet - or the control plane - can dial into a cluster.
2. So how does "start this job" arrive? The cluster dials out first. At startup, each cluster opens an outbound connection on port 443 to the SCC relay, forming a reverse tunnel.
3. The control plane sends instructions back down that already-open tunnel. It never initiates inbound.

Serverless reaches the same property differently

Serverless VMs also have no public IPs, and compute-to-control is always over the Microsoft backbone with TLS. There is no SCC relay because there is no VNet of yours.

Backbone vs Internet: The "Is Our Data On The Internet?" Talk

• Public internet - the shared, open roads anyone can travel; what auditors worry about.
• Microsoft backbone - Microsoft's private global network; Azure-to-Azure traffic stays on it.
• "Public IP" - here it just means an address reachable over the backbone, not necessarily exposed to the open internet.

The Microsoft backbone is Microsoft's private global network. Azure-to-Azure traffic rides it without ever touching the public internet.

A lot of Databricks traffic is already off the public internet by default:

• All classic compute-to-control-plane traffic rides the backbone - even with SCC disabled.
• All serverless traffic rides the backbone.

What "back-end Private Link removes the public-IP hop" means

The back-end path is compute → control plane (Door ②, the cluster dialing the SCC relay on 443). SCC and back-end Private Link remove public IPs from two different ends of that same connection:

With SCC alone, your VM has no public IP, but the address it dials is still the relay's public IP (over the backbone) - which is exactly why you keep the AzureDatabricks service tag in your egress allowlist. Back-end Private Link puts that relay behind a private IP in your own VNet, so the outbound 443 connection never targets a public address at all. That last public-IP hop is gone.

The Three Connectivity Paths

Secure a building and you ask exactly three questions: who comes in the front door, how staff reach head office, and how the loading dock is locked. Same three for Databricks - and almost every control protects one of them.

Interactive: click the boxes

Click each box to see what travels through that door and how it is secured.

How each door hardens

• Path ① (front door) is identical for classic and serverless - always to the control plane over HTTPS 443. Hardens from open public, to IP access lists, to front-end Private Link with public access disabled.
• Path ② (intercom) differs most by plane. Classic = SCC relay, outbound 443; back-end Private Link replaces the public-IP hop. Serverless = always backbone + TLS, no SCC.
• Path ③ (loading dock) is scrutinized most. Classic to ADLS hardens from public endpoint, to storage firewall + Service Endpoint (free, backbone-private), to Private Endpoint (private IP, per-GB cost). Serverless to ADLS uses NCC.

Deployment & Workspace Storage

• Resource group - an Azure folder that groups related resources together.
• Managed RG - the folder Databricks creates and runs for you; view it, but do not edit it (lock it).
• Metastore - the central catalog that decides who can see which data, one per region.
• Azure RBAC vs account admin - Azure roles create the workspace resource; a separate Databricks account admin governs the data. Being Azure Owner does not make you a Databricks admin.

Azure Databricks is a first-party Azure service (Microsoft.Databricks/workspaces) - sold, billed, and RBAC'd by Azure itself. One create lands resources in two resource groups.

Interactive: click the boxes

Click each box to see who owns it after a single workspace create.

The governance spine

The account to workspace to metastore spine is the backbone every later lesson hangs off:

• Account - identity, billing, Unity Catalog metastores. Holds many workspaces.
• Workspace - the runtime/collaboration unit, in one region.
• Metastore - account-level, regional governance root. One per region, shared by that region's workspaces.

Field Notes: Symptom to First Thought

Do not start by memorizing ports or service tags. Start by asking which plane, which direction, and which of the three doors is involved.

The seven questions

1. Which plane is the compute - classic (your VNet) or serverless (no VNet)?
2. Which direction does the connection point? (Compute always dials out.)
3. Is this traffic on the backbone or genuinely on the public internet?
4. Which of the three paths is involved - front door, intercom, or loading dock?
5. Where does the data actually live - UC + ADLS Gen2, or (wrongly) the DBFS root?
6. Which resource group owns this - my RG, or the locked managed RG?
7. Which role system is in play - Azure RBAC, or Databricks account admin?