Security: Encryption, Isolation & Compliance

8.1 · Encryption: TLS, CMK & Azure Key Vault

Encryption is on by default — TLS 1.2+ in transit, AES-256 at rest. CMK doesn't add a second safe; it swaps whose key opens the existing one from Microsoft's to yours in Key Vault. Switch the diagram between the three CMK features and the TLS-in-transit view; click any box for the "what + why."

Microsoft-managed vs Customer-Managed Keys

Illustrative config — CMK at workspace creation

Terraform (azurerm) — the encryption arguments. Verify exact names against current provider docs; full CMK + Key Vault IaC is the Stage 8 hands-on artifact.

resource "azurerm_databricks_workspace" "this" {
  sku = "premium"                                              # CMK requires Premium
  managed_services_cmk_key_vault_key_id = azurerm_key_vault_key.ms.id    # control-plane data
  managed_disk_cmk_key_vault_key_id     = azurerm_key_vault_key.disk.id   # classic VM disks
  customer_managed_key_enabled      = true   # DBFS-root CMK
  infrastructure_encryption_enabled = true   # DOUBLE encryption (deploy-time only)
}

Portal: Entra ID → Enterprise apps → AzureDatabricks · Key Vault (purge protection) → RSA key → IAM → Key Vault Crypto Service Encryption User to the app + storage MI · Create workspace → Encryption tab → Use your own key → paste the Key Identifier.

Uses, edge cases & limitations

• Uses: default TLS+at-rest answers "is it encrypted?" for free; CMK when an auditor requires customer key custody/rotation/revocation of control-plane data, workspace storage, or VM disks.
• Edge cases: CMK doesn't cover the ADLS lake; rotation needs the old key kept 24h; existing data isn't re-encrypted; managed-disk CMK doesn't apply to serverless (ephemeral disks).
• Limitations: all CMK = Premium + same-tenant Key Vault; managed-services CMK irreversible; key must be RSA, version explicit; double encryption deploy-time only.

8.2 · Compute Security & Isolation

The serverless trust question: "if compute runs in Databricks' account, what stops another tenant — or Databricks — from reading our data?" The answer is architecture, not your perimeter. Switch the diagram between the scoped-token data flow and the isolation layers; click any box.

Classic vs serverless isolation

Illustrative config — the UC scoped-credential chain

Terraform — core isolation (wipe, 1h token) is platform-enforced; you wire the plumbing.

resource "azurerm_databricks_access_connector" "uc" {   # the managed identity UC uses
  identity { type = "SystemAssigned" }
}
resource "azurerm_role_assignment" "uc_blob" {          # least privilege — NOT Owner/account key
  scope = azurerm_storage_account.adls.id
  role_definition_name = "Storage Blob Data Contributor"
  principal_id = azurerm_databricks_access_connector.uc.identity[0].principal_id
}
resource "databricks_external_location" "this" {        # binds credential to one path (isolation unit)
  url = "abfss://data@${azurerm_storage_account.adls.name}.dfs.core.windows.net/"
  credential_name = databricks_storage_credential.this.name
}

Portal: NSP → Create (region = workspace region) → Associate ADLS (Transition mode) → Inbound rule: Service Tag AzureDatabricksServerless.<region> · Access Connector → ADLS IAM → Storage Blob Data Contributor → External Locations bind the abfss:// path · Compute → access mode = Standard (Lakeguard).

Uses, edge cases & limitations

• Uses: approving serverless for a regulated customer; designing the access-connector → storage-credential → external-location chain; picking Standard mode for safe multi-user classic clusters.
• Edge cases: the 2026-06-09 firewall migration; NSP enforced mode breaks other services; direct abfss:// path access denied outside UC securables; Standard-mode capability gaps (no DBR ML/GPU/RDD/Scala) push some to Dedicated.
• Limitations: core isolation isn't tunable; the serverless service tag is for inbound rules to Azure Storage in the workspace region only; NSP Terraform coverage lags the Portal.

8.3 · Compliance: ESC, ESM & CSP

Three layers customers conflate, nested like Russian dolls: ESC (the paid add-on/bill), ESM (hardened image + agents, classic only), CSP (the umbrella posture + standards). CSP ⊃ ESM ⊃ hardened image. Switch the diagram between the nesting and the classic-plane impact; click any box.

ESC vs ESM vs CSP

Illustrative config — CSP + ESM + auto-update

Terraform (azurerm) — NOTE: only HIPAA / PCI_DSS / NONE here; other standards need Portal/CLI/PS/ARM.

resource "azurerm_databricks_workspace" "this" {
  sku = "premium"                                          # ESC requires Premium
  enhanced_security_compliance {
    automatic_cluster_update_enabled      = true            # permanent under CSP
    compliance_security_profile_enabled   = true            # CSP on (PERMANENT!)
    compliance_security_profile_standards = ["HIPAA", "PCI_DSS"]   # or ["NONE"]
    enhanced_security_monitoring_enabled  = true            # ESM agents + hardened image
  }
}
# On a restricted-egress VNet, also allow: Outbound TCP  -> * : 2443

Portal: workspace → Settings → Security & compliance → enable compliance security profile → pick standard(s) in-region or None → enable ESM → enable automatic cluster update. Confirm: a shield icon appears by the workspace name.

Uses, edge cases & limitations

• Uses: regulated workspaces (CSP + matching standard); high internal bar without attestation (CSP NONE); hardened image/monitoring on classic without permanence (ESM-only).
• Edge cases: permanence (delete+rebuild to revert); serverless region gaps; restricted-preview surface blocks brand-new previews; Gen2/Arm64 excluded; forgetting TCP 2443; AI features off by default.
• Limitations: Premium + ESC required & billed; ESM classic-only; Terraform standards subset (HIPAA/PCI_DSS/NONE); you own log review.

8.4 · Audit Logs, System Tables & Monitoring

One audit stream, two taps. System tables (system.access.audit) you query in SQL — the only tap carrying account-level events; Azure diagnostic settings export to your SIEM — workspace-level only. Switch the diagram between the two taps and the SIEM bridge; click any box.

Choosing your audit/monitoring path

Illustrative config — grant, alert query & SIEM export

SQL — least-privilege grant + a failed-login alert (the event_date predicate is required).

GRANT SELECT ON TABLE system.access.audit TO `secops-analysts`;

SELECT user_identity.email AS user_email, count(*) AS failed_logins
FROM system.access.audit
WHERE event_date >= current_date() - INTERVAL 1 DAY        -- prune partitions (REQUIRED)
  AND service_name = 'accounts'
  AND action_name IN ('login','aadBrowserLogin','tokenLogin')
  AND response.statusCode <> 200                            -- 200 = success; non-200 = the signal
  AND event_time >= current_timestamp() - INTERVAL 15 MINUTES
GROUP BY user_identity.email HAVING failed_logins >= 5;

Terraform — export workspace logs to Log Analytics for Sentinel.

resource "azurerm_monitor_diagnostic_setting" "adb_audit" {
  target_resource_id         = azurerm_databricks_workspace.this.id
  log_analytics_workspace_id = azurerm_log_analytics_workspace.sec.id
  enabled_log { category = "accounts" }      # auth, tokens, IP ACLs
  enabled_log { category = "unityCatalog" }  # grants, securable access
  # use a dynamic block — Databricks adds categories over time
}

Portal: Catalog Explorer → system → confirm system.access.audit · workspace resource → Monitoring → Diagnostic settings → + Add → categories → Log Analytics / Storage / Event Hub · SQL → Alerts → Create → paste query → condition → schedule → notifications.

This module ships a runnable companion: audit_monitoring.sql (grants + failed-login, sensitive-table, privilege-change, ESM-agent and cost-anomaly queries).

Decision guide — what an architect recommends

Customer Q&A — what the security team actually asks

Common asks & the answer

What breaks in the field — and the FIRST check

Three questions to ask before recommending

1. Is there a mandate for customer-held keys (CMK), or do encryption defaults suffice?
2. Must you process data under a named standard — and is the standard supported on your plane × region (CSP is permanent)?
3. Do you have a mandated central SIEM / >365-day / sub-minute requirement (then both taps), or is in-platform investigation enough?