FDE Field Playbooks

10.1 · Compute & networking startup failures

A starting cluster is a delivery truck leaving the depot — it needs a parking spot (vCPU quota), two licence plates (host + container IP), an open gate with the right pass (NSG/delegation), and a working road to head office (UDR/NAT → SCC relay). Switch the diagram between the five failure families and click a box to see what it is, why it's there, and what breaks.

Illustrative config — the forced-tunnel UDR that keeps the relay reachable

Terraform — keep control-plane traffic on the backbone via SERVICE TAGS; send the rest to the hub firewall. Missing the first routes = bootstrap timeout. Full module is the owning stage's hands-on artifact.

resource "azurerm_route" "adb_cp" {            # SCC relay + webapp + API
  address_prefix = "AzureDatabricks"           # SERVICE TAG, not a CIDR (IPs rotate)
  next_hop_type  = "Internet"                  # stays on the Microsoft backbone
}
resource "azurerm_route" "default_to_fw" {     # everything else -> hub firewall
  address_prefix         = "0.0.0.0/0"
  next_hop_type          = "VirtualAppliance"
  next_hop_in_ip_address = "10.100.0.4"        # hub Azure Firewall private IP
}
# Also required: routes for "Storage" (443) and "EventHub" (9093).

Portal: termination reason at Compute → cluster → Event log + Driver logs. Failed deploy → RG → Deployments (ARM error). Quota → Subscription → Usage + quotas. IPs → VNet → Subnets → Available IPs.

10.2 · Connectivity — front-end, Private Link, DNS & SCC

The workspace URL is a phone number, DNS is the phonebook, Private Link is a private unlisted line. ~80% of "Private Link is broken" tickets are really "the phonebook gives out the wrong number" — so check DNS first. Switch the diagram between the four paths and click a box.

Illustrative config — the probe + the forwarding fix

nslookup adb-1111111111111111.1.azuredatabricks.net   # from a jump-box in the VNet
#   GOOD: CNAME -> ...privatelink.azuredatabricks.net, Address 10.x.x.x
#   BAD : a public IP (20.x / 40.x) -> Private DNS zone not in use

az network private-dns link vnet list \
  --resource-group dns-rg --zone-name privatelink.azuredatabricks.net -o table

Fix: conditional-forward *.azuredatabricks.net, *.privatelink.azuredatabricks.net, *.databricksapps.com → Azure DNS 168.63.129.16. Portal: Private DNS zone → Virtual network links / Recordsets; workspace → Networking → Private endpoint connections (Approved).

10.3 · Storage & serverless access

Every storage read passes two turnstiles in order: the network gate (storage firewall — did the packet reach the building?) then the identity gate (RBAC + Unity Catalog — can you open this cabinet?). A 403 is just "a turnstile said no." Switch the diagram between classic and serverless — they take different doors.

Illustrative config — wire the identity gate (the most common 403)

-- UC: credential -> external location -> grant. The GRANT is the missing line in many 403s.
CREATE STORAGE CREDENTIAL adls_cred WITH AZURE_MANAGED_IDENTITY
  ACCESS_CONNECTOR_ID = '/subscriptions/<sub>/resourceGroups/adb-rg/providers/Microsoft.Databricks/accessConnectors/adb-ac';
CREATE EXTERNAL LOCATION lake_ext URL 'abfss://data@mydatalake.dfs.core.windows.net/'
  WITH (STORAGE CREDENTIAL adls_cred);
GRANT READ FILES ON EXTERNAL LOCATION lake_ext TO `data-engineers`;

# Azure: the access-connector identity needs the DATA-PLANE role (not "...Reader").
az role assignment create --assignee-object-id <MI-principal-id> \
  --assignee-principal-type ServicePrincipal --role "Storage Blob Data Contributor" \
  --scope ".../storageAccounts/mydatalake"

Portal: storage → Networking (firewall / PE connections / resource-instance rule for Microsoft.Databricks/accessConnectors); Account console → NCC (create → add PE rule → bind → approve PE on storage → wait ~10 min → restart serverless).

10.4 · Customer deployment patterns & the decision flow

Three reference architectures are three rungs of a security ladder, not three products. Start every customer on the bottom rung and climb only as far as their compliance mandate forces — each rung up bolts on billable Azure parts and DNS/firewall ops. Switch the diagram between the rungs.

The feature matrix (the interview answer)

Illustrative config — the delta from baseline to Isolated

resource "azurerm_databricks_workspace" "isolated" {
  sku                                   = "premium"               # all Private Link needs Premium
  public_network_access_enabled         = false                   # <-- Isolated: reject public
  network_security_group_rules_required = "NoAzureDatabricksRules" # <-- back-end PL needs this
  custom_parameters { no_public_ip = true /* + VNet/subnet/NSG associations */ }
}
# + databricks_ui_api PE (back-end) and databricks_ui_api + browser_authentication PE
#   (front-end, in the TRANSIT VNet) complete the Isolated inbound path.

Portal: Networking → SCC=Yes, own VNet=Yes (all rungs); Required NSG rules = AllRules (Managed) / NoAzureDatabricksRules (back-end PL); Public network access = Enabled (Managed/Hardened) / Disabled (Isolated).

10.5 · Diagnostics, support & escalation

You're an air-crash investigator, not a witness. Your black boxes — system tables, VNet flow logs, diagnostic-settings export — only have a recording if you switched them on before the crash. Switch the diagram between the diagnostic layers and the escalation flow; click a box for what it answers and its retention.

Illustrative config — read the network-denial black boxes

SELECT event_time, source.ip AS src_ip, authenticated_as, policy_outcome, rule_label
FROM   system.access.inbound_network                       -- 30-day retention
WHERE  event_time >= current_timestamp() - INTERVAL 2 HOUR ORDER BY event_time DESC;

SELECT event_time, destination_type, destination,
       dns_event.rcode AS dns_rcode, storage_event.rejection_reason
FROM   system.access.outbound_network                      -- DNS vs IP vs STORAGE denial
WHERE  event_time >= current_date() - INTERVAL 1 DAY ORDER BY event_time DESC;

# Enable VNet flow logs BEFORE incidents — a 'D' record names the rule + 5-tuple that dropped the packet.
az network watcher flow-log create --resource-group rg-network --name adb-vnet-flowlog \
  --location eastus --vnet adb-vnet --storage-account stflowlogseastus \
  --traffic-analytics true --workspace <log-analytics-resource-id>

Status first: status.azuredatabricks.net + Service Health + Resource health. Escalate: Help + support → Create a support request → Technical → Azure Databricks → the workspace's subscription (wrong subscription = the engineer can't see your resources).

Decision guide — what an architect recommends

Customer Q&A — what the security/network team actually asks

Common asks & the answer

What breaks in the field — and the FIRST check

Four questions to ask before recommending an architecture

1. Is there a regulatory mandate that traffic never touches the public internet?
2. Do you need data-exfiltration protection (block egress to arbitrary internet)?
3. Do you need ingress allowlisting + egress control + private data, but public login is OK?
4. Is RFC 1918 scarce or will a workspace autoscale to hundreds of nodes (overlay IP-exhaustion)?