Databricks DE · Topic 8.1

Unity Catalog — Namespace & Governance

One governance layer over all data & AI: a metastore→catalog→schema→object hierarchy, the 3-level name catalog.schema.object, Volumes for files, plus built-in discovery, lineage, and audit. This deep dive walks each with the code.

catalog.schema.objectinformation_schemaVolumeslineage + audit

The one-liner

Unity Catalog governs every asset under metastore → catalog → schema → object, addressed by the 3-level name catalog.schema.object. Volumes govern files; lineage & audit are automatic (queryable as system tables).
📚 Analogy: a city library system — metastore = the system, catalogs = branches, schemas = sections, tables/volumes = books/media. The 3-level name is the call number; lineage is the citation trail; audit is the checkout log.

The object hierarchy — click a level

▸ metastore → catalog → schema → object

🏛️ metastore
📁 catalog  (prod)
📂 schema  (sales)
📄 object  (orders / landing volume / fn / model)

1 · The 3-level namespace

Create the hierarchy, then address objects fully-qualified or set a default catalog+schema for short names. Common pattern: catalog per environment/domain.

CREATE CATALOG IF NOT EXISTS prod;
CREATE SCHEMA  IF NOT EXISTS prod.sales;

SELECT * FROM prod.sales.orders;     -- fully qualified
USE CATALOG prod; USE SCHEMA sales;     -- then short names resolve
SELECT * FROM orders;                 -- → prod.sales.orders

Inspect metadata via information_schema

SELECT table_name, table_type FROM prod.information_schema.tables WHERE table_schema = 'sales';
SELECT column_name, data_type FROM prod.information_schema.columns
WHERE table_schema = 'sales' AND table_name = 'orders';

2 · Volumes — govern files (not tables)

The UC-governed place for files; access at /Volumes/<catalog>/<schema>/<volume>/…. Managed = UC owns storage; external = your path under a UC external location. Replaces DBFS mounts.

CREATE VOLUME prod.sales.landing;                         -- managed
CREATE EXTERNAL VOLUME prod.sales.archive
  LOCATION 's3://acme-lake/sales/archive/';              -- external
# Read/write files through the /Volumes path — governed like any UC object
dbutils.fs.ls("/Volumes/prod/sales/landing/")
df = spark.read.json("/Volumes/prod/sales/landing/clickstream.json")
df.write.format("delta").saveAsTable("prod.sales.bronze_clicks")

3 · Built-in governance — click each

Discovery — Catalog Explorer + search to find/inspect assets; information_schema & COMMENT/TAGS make them searchable.
Lineage — automatically tracks source → tables → dashboards, down to column level. Query via system.access.column_lineage.
Audit — the system.access.audit system table records all data access & system activity for compliance/security.
Volumes — govern non-tabular files (images, JSON, models), managed or external — instead of legacy DBFS mounts.

4 · Tags, lineage & audit as code

-- Document + tag for search/governance
COMMENT ON TABLE prod.sales.orders IS 'Curated order facts (gold).';
ALTER TABLE prod.sales.orders SET TAGS ('domain'='sales', 'pii'='false');
ALTER TABLE prod.sales.orders ALTER COLUMN email SET TAGS ('pii'='true');
-- Who accessed what (audit) + column lineage — via system tables
SELECT event_time, user_identity.email, action_name FROM system.access.audit
WHERE action_name = 'getTable' AND event_date >= current_date() - 7;
SELECT * FROM system.access.column_lineage WHERE target_table_name = 'orders';

Why it matters

🧾 Real-world use case: prod.finance.invoices is governed once — analysts discover it in Catalog Explorer, lineage shows it feeds the revenue dashboard, and audit logs record every access for compliance.

Uses, edge cases & limitations

Common gotchas