Databricks · One-Pager · GenAI / Eval

GenAI Evaluation

MLflow 3's framework for measuring GenAI/agent quality with LLM-as-a-judge scorers, evaluation datasets, human labeling, and continuous production monitoring — all on top of MLflow Tracing.

GA · MLflow 3 mlflow.genai.evaluate() Monitoring = Beta Verified June 2026 docs

In one line

GenAI Evaluation is mlflow.genai.evaluate() running scorers (LLM-as-a-judge plus code-based) over an evaluation dataset or live traces to score quality — the Mosaic AI Agent Evaluation SDK now folded into Databricks-managed MLflow 3.

Why it matters

GenAI outputs are non-deterministic, so you can't unit-test them — you need graded judgment. Evaluation gives a repeatable quality signal you can use to compare agent/prompt versions, catch regressions before release, and keep watching quality in production with the same scorers.

Core concepts

LLM-as-a-judge scorers

A judge LLM grades each output. Built-in: Correctness, Guidelines, RelevanceToQuery, Safety, RetrievalGroundedness, RetrievalRelevance — most need no ground truth.

Custom judges & scorers

make_judge() builds a custom LLM judge from natural-language instructions; the @scorer decorator wraps any deterministic Python check (regex, latency, exact match).

Evaluation datasets

UC-backed (Delta) sets of records — each an inputs dict plus optional expectations ground truth. Built from humans, documents, or production traces.

Human feedback & labeling

The Review App lets domain experts (SMEs) attach feedback to traces via label schemas in labeling sessions; those labels seed eval datasets and align judges.

Trace-based evaluation

Every eval row produces an MLflow Trace; scorers read its spans (retriever, tool, LLM), so the same judges work offline and on captured production traces.

Production monitoring (Beta)

Register a scorer to an experiment and start() it with ScorerSamplingConfig(sample_rate=...) to grade a sample of live traces continuously.

How it works

Click a stage

1 · Dataset + traces 2 · Scorers (judges) 3 · Metrics + results 4 · Iterate / monitor

Dataset + traces — an eval dataset (inputs + optional expectations) or captured traces feed evaluate(); predict_fn generates outputs and a trace per row.

Minimal example

Python · mlflow[databricks]>=3.1import mlflow
from mlflow.genai.scorers import Correctness, Guidelines, Safety

results = mlflow.genai.evaluate(
    data=eval_dataset,            # rows: {"inputs": {...}, "expectations": {...}}
    predict_fn=my_agent,          # receives **inputs, returns outputs; auto-traced
    scorers=[
        Correctness(),               # LLM judge vs expectations.expected_facts
        Guidelines(name="tone", guidelines="Response must be concise and polite"),
        Safety(),                    # no ground truth needed
    ],
)
# results.metrics -> aggregate scores; per-row feedback attached to each trace

Gotchas & limits

Use mlflow.genai.evaluate(), not the old mlflow.evaluate() — and data rows use the nested {"inputs": {...}} shape, not flat columns.
Judges cost money and vary: every scorer is an extra LLM call, and the same output can score differently across runs — sample, don't grade 100% in production, and align judges to SME labels before trusting them.
Some scorers need specific inputs: Correctness/ExpectationsGuidelines require expectations; RetrievalGroundedness needs a RETRIEVER span in the trace.
Production monitoring is Beta (June 2026), admin-gated via the Previews page, and needs a SQL warehouse configured for trace storage.

Interview soundbites

Say this out loud

"GenAI eval is mlflow.genai.evaluate() with scorers — built-in LLM judges like Correctness, Safety, and RelevanceToQuery, plus custom judges via make_judge() and code scorers via @scorer."
"The same scorers run three places: offline on an eval dataset, on captured traces, and continuously on a sample of production traces — so dev and prod quality use one definition."
"Most judges don't need ground truth; only Correctness and the expectations-based ones do. Humans add labels through the Review App to build datasets and align judges."

References

Verified against docs.databricks.com · June 2026