Databricks GenAI governance and evaluation
Governance (8%) and Evaluation and Monitoring (12%) are the two smallest sections of the Databricks Generative AI Engineer Associate exam — 20% between them, roughly 9 of the 45 scored questions. Small, but cheap to learn and easy marks if you spend an evening on them.
Governance (8%)
Around four questions. Four ideas cover most of it.
Access control through Unity Catalog. Who can see which data, which models, and which indexed content. The recurring scenario: retrieval must respect the permissions the user already has, so people cannot surface documents through an assistant that they could not open directly.
Guardrails. Constraints on what the application will do — blocking disallowed topics, filtering unsafe output, and restricting the scope of what it will discuss. Guardrails sit around the model because prompting alone cannot fully control behaviour.
Sensitive data. What may be sent to a model, what may be indexed, and what should be excluded or masked before it ever reaches the pipeline.
Lineage and auditability. What a model was built from, what data an application can reach, and a record of use. Unity Catalog provides this.
Evaluation and monitoring (12%)
Around five questions, and conceptually the more interesting section.
Why evaluation is hard here
A classifier has a right answer. A generative application usually does not, so “accuracy” is not available. You measure different properties instead:
| Measure | The question it answers |
|---|---|
| Groundedness | Is the answer supported by the retrieved content? |
| Relevance | Does the answer address what was asked? |
| Correctness | Is it factually right, against a reference? |
| Retrieval quality | Was the right content retrieved at all? |
| Safety | Did it produce anything it should not? |
Groundedness is the one to know cold. It is the measure that catches the characteristic failure of a RAG system — a fluent, confident answer not supported by the sources.
Evaluation sets
A fixed set of questions with expected answers or reference content, run against every version. Fixed matters: if you change both the application and the test questions, you cannot attribute the difference.
The diagnostic habit: when quality drops, check retrieval quality separately from answer quality. If the right content was not retrieved, the model was never given a chance, and fixing the prompt or the model will not help.
Monitoring in production
- Answer quality over time, sampled and reviewed.
- Latency across the chain’s steps.
- Token consumption and cost per query.
- Corpus drift — the documents change, and an index that is not refreshed quietly goes stale.
- What users actually ask, which often differs from what you designed for.
Sample questions
Question 1. An assistant produces a fluent, confident answer that is not supported by any retrieved document. Which measure is designed to detect this?
- A. Average latency per query
- B. Groundedness
- C. Relevance
- D. Tokens consumed per query
Show answer
Answer: B
Groundedness measures whether a response is supported by the retrieved source content, which is exactly the failure described. Latency measures speed, relevance only checks the answer addresses the question, and token count measures consumption.
Question 2. After a change, answer quality has dropped. The team must determine whether the cause is retrieval or generation. What should they measure?
- A. Retrieval quality separately from answer quality, using a fixed evaluation set
- B. Overall user satisfaction scores
- C. End-to-end latency per request
- D. Token cost per query before and after the change
Show answer
Answer: A
Measuring retrieval quality separately establishes whether the correct content was returned at all, which distinguishes a retrieval problem from a generation problem. Overall satisfaction does not isolate the stage, latency measures speed, and token cost measures spend.
Question 3. An organisation requires that an assistant only retrieve documents the requesting user is permitted to access, with an auditable record. Which capability provides this?
- A. A content guardrail on the model’s output
- B. MLflow model registry permissions
- C. Unity Catalog access controls over the indexed data
- D. An instruction in the system prompt to respect permissions
Show answer
Answer: C
Unity Catalog centralises access control and lineage over data and models, which is what permission-scoped retrieval with auditability requires. A guardrail filters content rather than enforcing per-user permissions, MLflow manages model lifecycle, and prompt instructions cannot enforce access control.
What to practise
Build a ten-question evaluation set for a RAG application you have built, and score both retrieval and answers. Then degrade the index deliberately and re-run it. Watching retrieval quality fall while answers remain fluent makes both of these sections concrete — and it is exactly the reasoning the questions test.