RAG explained: chunking, embeddings, retrieval
Retrieval-augmented generation (RAG) is a pattern in which an application first searches your own content for passages relevant to a question, then gives those passages to a large language model together with the question, so the model answers from your material instead of from memory. The exam decision rule: if the information changes or must be grounded in your documents and cited, use RAG; if the model’s behaviour or style must change, that is fine-tuning. RAG needs no retraining when the content changes, you only update the index.
The pipeline, stage by stage
RAG has two phases. Indexing happens ahead of time; the query path runs on every question.
Indexing (done once, then kept fresh)
- Ingest and clean. Pull documents from their sources and extract text. Scanned PDFs need OCR first.
- Chunk. Split documents into passages. Models accept a limited number of input tokens, and one vector for a whole page represents a multi-topic page poorly. Microsoft’s guidance for Azure AI Search suggests starting around 512 tokens per chunk with roughly 25% overlap, and says the best values depend on the content. Overlap exists so a sentence cut at a boundary still appears whole in one chunk.
- Embed. An embedding model turns each chunk into a vector of numbers so that passages with similar meaning sit close together. AWS describes embeddings as numeric representations that capture semantic meaning.
- Store. Vectors, the original text and metadata (source, product, date, access group) go into a vector index or a search service with vector support.
Query path (every question)
- Retrieve. The question is embedded the same way and the index returns the nearest chunks. Hybrid search combines keyword and vector matching for better recall, and a semantic ranker can re-score the results.
- Augment. The top chunks are placed into the prompt with instructions such as “answer only from the context below”.
- Generate. The model writes the answer, ideally with citations to the chunks used. That tie to source text is what the exam calls grounding.
Databricks documentation gives the same shape in three words: retrieval, augmentation, generation.
A small worked example
An HR assistant must answer questions from policy PDFs that are revised twice a year. Employee asks: “How many days of parental leave do contractors get?”
The question is embedded, and the index returns three chunks: the contractor section of the leave policy, the general parental leave paragraph, and an unrelated expenses paragraph. The prompt carries the first two plus the instruction to answer only from them and to say so if the answer is missing. The model replies with the contractor entitlement and names the policy section.
When HR publishes a new policy version, the team re-indexes the changed file. Nobody retrains anything. Had the team instead fine-tuned a model on the old PDFs, the model would keep quoting the old numbers until retrained, and could not show where an answer came from.
Which stage to fix for which symptom
Most RAG failures are retrieval failures, so the exam habit is: check retrieval before touching the model or the prompt.
| Symptom | Likely stage | Fix |
|---|---|---|
| Answer is fluent but names the wrong product or policy | Retrieval | Better chunking, metadata filters, hybrid search, reranking |
| Answer misses a fact that is in the document | Chunking or retrieval | Smaller or overlapping chunks; retrieve more candidates |
| Answer contains facts that are in no document | Generation | Stronger grounding instruction, lower temperature, require citations, add guardrails |
| Users see documents they should not | Security | Filter results by the caller’s permissions at query time |
| Answers are stale | Indexing | Incremental re-indexing on a schedule or trigger |
| Slow and expensive responses | Augmentation | Return fewer, more relevant chunks; cap result count |
Azure AI Search documentation lists access control (security), token constraints and response time among the core challenges of RAG, which matches this table.
One product note: Azure AI Search now offers agentic retrieval, where an LLM plans and splits a complex question into subqueries. Microsoft labels the LLM-based query planning as preview, with the classic single-query pattern as the generally available option. Treat the details as subject to change and check the current page before an exam that names it.
Which exams test this
| Exam | How it appears | On this site |
|---|---|---|
| AIF-C01 | Applications of Foundation Models (28%): defining RAG and its business uses (Amazon Bedrock Knowledge Bases), vector database services, and the cost trade-off between RAG, fine-tuning and other customisation; Security, Compliance, and Governance (14%): RAG grounding against hallucination | Foundation model applications |
| AI-103 | Implement generative AI and agentic solutions (30–35%): implementing RAG in an application; Implement information extraction solutions (10–15%): retrieval and grounding pipelines, including ingestion with OCR and hybrid and vector search | Information extraction |
| AI-300 | Optimize generative AI systems and model performance (10–15%): tuning similarity thresholds, chunk sizes and retrieval strategies, choosing embedding models, hybrid search, and measuring RAG with relevance metrics and A/B tests | GenAI optimization |
| Databricks GenAI Engineer | Data Preparation (14%): chunking strategies, filtering source content, evaluating retrieval and re-ranking; Assembling and Deploying Applications (22%): the building blocks of a RAG application and a Vector Search index | Design and data preparation |
| DP-800 | Implement AI capabilities in database solutions (25–30%): RAG use cases, building the prompt with sp_invoke_external_rest_endpoint, converting rows to JSON and reading the model’s response, on top of embeddings and hybrid search | AI capabilities |
| AI-200 | Develop AI solutions by using Azure data management services (25–30%): RAG patterns with metadata filters on PostgreSQL with pgvector, plus vector search in Cosmos DB and Azure Managed Redis | Data management |
| MLA-C02 | Data Preparation (28%): chunking and metadata extraction for RAG; Model and FM Development (24%): choosing a RAG architecture and monitoring retrieval accuracy; Deployment and Orchestration (24%): Bedrock knowledge bases, retrieval pipelines, reranking and knowledge base refresh | Data preparation |
| AB-731 | Identify the business value of generative AI solutions (35–40%): grounding requirements and how RAG is used, at a business rather than build level | Business value of generative AI |
| AB-620 | Integrate and extend agents in Copilot Studio (40–45%): Azure AI Search as a knowledge source and generative answers grounded through Azure AI Search with Foundry. RAG appears here as product configuration, not as a named concept | Integrate and extend |
| Agentforce Specialist | Data 360 Fundamentals (20%): chunking, indexing and retrievers; Prompt Engineering (20%): choosing a grounding technique | Data 360 |
| CY0-001 | Basic AI Concepts Related to Cybersecurity (17%): objective 1.2 lists RAG, vector storage and embeddings under data security for AI. A minor topic, seen from the security angle | CY0-001 basic AI concepts related to cybersecurity explained |
How exam questions are built around RAG
The scenario is nearly always a symptom plus a constraint, and four wrong options that fix a different layer. The recurring traps:
- RAG or fine-tuning. Words like current, frequently updated, cite sources, proprietary documents mean RAG. Consistent tone, domain vocabulary, output format mean fine-tuning. Prompting alone is the answer only when no private or changing data is involved.
- Fix the wrong layer. A larger model, a lower temperature or a longer prompt will not repair wrong chunks. If the right passage never reaches the model, the model cannot answer correctly.
- Permissions. “Users must only see content they are allowed to open” means filtering at retrieval time, not hiding the answer afterwards.
- Hybrid over pure vector. Exact identifiers, part numbers and rare terms match poorly by meaning alone, so keyword plus vector is the answer when the stem mentions them.
- Groundedness versus relevance. Relevance measures whether the answer addresses the question, retrieval quality whether the right content was found, and groundedness whether the answer is supported by that content. They are separate metrics.
Practice questions
Question 1. A support assistant retrieves five chunks per question from product manuals. Users report that answers read well but often describe the wrong product model. Logs show that the retrieved chunks frequently come from other models' manuals. What is the most effective change?
- A. Switch to a larger language model
- B. Lower the model temperature to 0
- C. Add product-model metadata to the chunks and filter or boost on it in hybrid search
- D. Fine-tune the model on the manuals
Show answer
Answer: C
The fault is in retrieval: the wrong chunks reach the model, so improving retrieval with product-model metadata filters and hybrid search addresses the cause. A larger model, a lower temperature or a longer system prompt still receive the wrong context and will keep answering confidently from it. Fine-tuning does not fix which passages are retrieved.
Question 2. A company wants a chatbot that answers from its internal policy documents. The policies are revised every month and answers must show which document supports them. Which approach fits best?
- A. Retrieval-augmented generation over an index of the policies
- B. Fine-tune a model each month on the new policies
- C. Prompt engineering with no external data
- D. Pre-train a new language model on the policies
Show answer
Answer: A
RAG retrieves the current policy passages at question time, so a monthly revision only needs re-indexing, and the retrieved chunks give a source to cite. Fine-tuning bakes content into the model, goes stale until retrained and cannot point to a source. Prompt engineering alone has no access to the private documents. Pre-training a new model is far beyond the requirement.
Go deeper
- RAG and generative AI in Azure AI Search: classic versus agentic retrieval, hybrid search and security trimming.
- Chunk documents in Azure AI Search: chunking techniques, size and overlap.
- Understanding Retrieval Augmented Generation (AWS Prescriptive Guidance) and Databricks RAG overview.
- Related topics: fine-tuning vs RAG vs prompting and AI agents explained.