Databricks Genie

The one-liner

Ask in English — and see why curation matters

▸ toggle curation to compare the answer

How it works — deep dive

1 · The Genie space — scope + datasets

Mechanism: a space is bound to a set of UC tables/views and runs on a SQL Warehouse — the unit of scope, curation & permission. Trade-off: too many/messy tables dilute accuracy; aim at a focused, well-named gold domain.

-- A space points at curated gold objects; make columns self-describing.
CREATE OR REPLACE VIEW gold.sales_fact AS
SELECT order_id, region, net_revenue, fiscal_quarter, order_date
FROM prod.sales.orders_enriched;
COMMENT ON COLUMN gold.sales_fact.net_revenue IS 'Revenue after returns/discounts (finance definition)';

2 · Natural language → SQL

Mechanism: Genie maps the question to SQL using schemas + column comments + curation, runs it, and returns the SQL and result so users verify & refine. Trade-off: ambiguity → plausible-but-wrong query; fix with curation, not luck.

-- "revenue by region last quarter" → Genie generates (and shows):
SELECT region, sum(net_revenue) AS revenue
FROM gold.sales_fact
WHERE fiscal_quarter = '2026-Q1'
GROUP BY region ORDER BY revenue DESC;

3 · Curation & trust — instructions, example SQL, metrics

Mechanism: curate with general instructions, example SQL, certified/sample questions, and defined joins & metrics so "revenue" resolves to one definition. Trade-off: ongoing upkeep — but it's what makes answers trustworthy.

-- Encode the canonical metric as a governed object everyone (incl. Genie) reuses:
CREATE OR REPLACE VIEW gold.active_customers AS
SELECT * FROM prod.sales.customers
WHERE status = 'ACTIVE' AND last_seen >= current_date() - INTERVAL 90 DAYS;
-- Space instruction: "'active customer' = rows in gold.active_customers."

4 · Governance — UC grants, row filters, column masks

Mechanism: Genie queries run as the asking user through Unity Catalog, so existing grants, row filters, and column masks apply automatically — it can't surface data the user couldn't query. Trade-off: users get different rows by grant (correct, but explain it).

-- Grant the curated domain; an existing row filter (8.3) still applies in Genie.
GRANT SELECT ON VIEW gold.sales_fact TO `sales-ops`;
-- ALTER TABLE prod.sales.orders_enriched
--   SET ROW FILTER catalog.sec.region_filter ON (region);  -- scopes rows per user

5 · Genie is programmable — the Conversation API

Mechanism: drive a space from code — start a conversation, send follow-ups, poll the message until done. Trade-off: start a new conversation per session (reuse bleeds context); poll every 1–5 s, cap ~10 min. ⚠️ verify the API version against docs.

# Start a conversation in a space (verified path, API 2.0):
curl -X POST \
  https://<host>/api/2.0/genie/spaces/<space_id>/start-conversation \
  -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  -d '{"content": "What was revenue by region last quarter?"}'

# Follow-up: POST .../spaces/<space_id>/conversations/<conversation_id>/messages
# Poll the message until status = COMPLETED / FAILED / CANCELLED.

6 · Genie vs a generic LLM chatbot

Mechanism: a generic chatbot answers from training data; Genie is grounded — it answers only by generating SQL over your governed tables, shows the query, and respects UC. Trade-off: it won't answer beyond its data (by design); review SQL for key decisions.

The Genie family — click each

Why it matters

• Democratizes data — non-SQL users self-serve — while staying inside UC governance.
• "How do non-technical users query the lakehouse safely?" → a Genie space, and curation drives accuracy.
• Programmable via the Conversation API — embed Genie in apps/agents, not just the chat box.

Uses, edge cases & limitations

• Uses: self-serve analytics, ad-hoc Q&A over curated domains, reducing the analyst queue, embedding Q&A via the API.
• Edge cases: accuracy depends on curation — thin spaces give vague/wrong answers; ambiguous terms ("active user") need a defined metric; per-user governance means different users get different rows.
• Limitation: assistive — review generated SQL for key decisions; best over clean, well-named gold tables; availability varies by region.

Common gotchas

• ❌ Expecting great answers from an uncurated space.
• ❌ Pointing Genie at messy bronze — aim it at clean gold.
• ❌ Treating output as always-correct — verify the SQL for key numbers.
• ❌ Confusing Genie (business Q&A) with Genie Code (build code/pipelines).
• ❌ Reusing one API conversation across unrelated sessions — start a new one to avoid context bleed.