Fine-tuning vs RAG vs prompting
Prompting, retrieval-augmented generation (RAG) and fine-tuning are three ways to make a general model fit a specific job, and they fix three different problems. Prompting changes the instructions for one request. RAG changes what the model can see by fetching your content at question time. Fine-tuning changes the model itself by training it on examples. The decision rule: if the information changes or must come from your documents, use RAG; if the behaviour must change (tone, format, a narrow skill), consider fine-tuning; if neither is true, start with a better prompt, because it is the cheapest option and the first one to try.
What each option actually changes
| Prompting | RAG | Fine-tuning | |
|---|---|---|---|
| What changes | The instructions and examples in the request | The context passed to the model on each request | The model’s weights |
| Fixes | Wrong format, vague output, wrong role or tone | Missing, private or outdated facts; need for citations | Consistent style, strict output structure, a narrow task a small model must do well |
| When your content changes | Edit the prompt | Update the index | Retrain and redeploy |
| Up-front effort | Minutes | A pipeline: chunking, embeddings, an index | A curated, labelled dataset and a training run |
| Cost shape | Tokens per call | Retrieval plus longer prompts on every call | Training cost, then a hosted custom model |
| Auditability | Visible prompt | You can see which passages were retrieved | Hard to trace why the model answered as it did |
The three are not exclusive. A production system often has a careful prompt, retrieval for facts, and occasionally a fine-tuned model underneath. The exam question is almost always “which one first” or “which one fixes this symptom”. How RAG itself works is covered in RAG explained; this page is about when to pick it.
The order in which to try them
The usual sequence is prompting, then RAG, then fine-tuning, moving to the next only when the previous one has plateaued. The reasons are practical: each step costs more, takes longer to change, and is harder to undo.
- Prompt first. Give a role, state the task, show one or two examples (few-shot), and specify the output format. Few-shot examples condition the model for that single call only; nothing is learned permanently. If the output is merely badly formatted, this is the fix.
- Add retrieval when the model lacks the facts. A model cannot answer from a policy it never saw, however well it is prompted. Supplying the passages in the prompt grounds the answer and lets you cite them.
- Fine-tune when behaviour is still wrong. Microsoft’s Foundry documentation lists what fine-tuning can help with: accuracy on a task, a consistent tone, format or schema, better tool use, and a smaller model that meets the task at lower cost. It names three methods: supervised fine-tuning (SFT) when you can demonstrate the desired responses, preference fine-tuning with DPO when quality depends on preferences such as tone, and reinforcement fine-tuning when several answers are possible and a grader can reliably score them. The same page states that fine-tuning does not replace retrieval for current information. Amazon Bedrock lists supervised fine-tuning, reinforcement fine-tuning and distillation as its customisation methods.
A related point that exams like: fine-tuning is not a good way to load a changing knowledge base. Facts baked into weights go stale, cannot be cited, and need another training run to correct.
A worked example
A law firm wants an assistant for staff. Three requirements arrive together.
- “Answers must reflect our current leave policy, which HR updates every quarter.” The facts change and must come from the firm’s documents, so this is RAG. Retraining every quarter would be slow and would still not give citations.
- “Every reply must open with a one-line summary and end with a reference list in a fixed layout.” This is format. Try a prompt with an example of the layout first. If the model still drifts after the prompt is tightened, a small fine-tune on a few hundred approved replies is the next step.
- “Replies should read like our house style, which thousands of approved letters demonstrate.” Style is behaviour, and the firm has plenty of examples, so this is the legitimate case for fine-tuning.
The result is a hybrid, and each part is justified by its own symptom. If a requirement can be solved earlier in the ladder, the later step is the wrong answer.
Which exams test this
| Exam | How it appears | Where on this site |
|---|---|---|
| AIF-C01 | Domain 3, Applications of Foundation Models (28%): cost trade-offs between pre-training, fine-tuning, in-context learning, RAG and distillation, plus prompt engineering techniques and the fine-tuning process | Foundation model applications explained |
| MLA-C02 | Domain 2, ML Model and FM Development (24%): fine-tuning strategies for pre-trained FMs, RAG architecture patterns, and customisation by task-specific prompt engineering or fine-tuning | Model development explained |
| AB-731 | Business value of generative AI (35–40%): fine-tuned versus pretrained models, the impact and techniques of prompt engineering, grounding requirements and how RAG is used | Generative AI business value explained |
| AI-300 | Optimize generative AI systems and model performance (10–15%): tune RAG retrieval, then advanced fine-tuning methods and synthetic training data; each technique is tested in depth rather than the choice between them | GenAI optimization explained |
| AI-103 | Implement generative AI and agentic solutions (30–35%): implement RAG in an application and tune generation with prompt engineering and model parameters; fine-tuning is not named in the outline | Generative AI and agents explained |
| Databricks GenAI Engineer | Design Applications (14%) and Data Preparation (14%): design prompts for a required output and build the RAG side (chunking, source documents, retrieval); fine-tuning is not an objective, so only the prompting-versus-RAG half applies | Design and data preparation explained |
| AB-100 | Plan AI-powered business solutions (25–30%): determine when custom AI models should be created and when customised small language models fit; a minor, architect-level view of the same decision | Plan AI solutions explained |
How exam questions are built around it
Scenarios hide the answer in one or two constraint words.
- Changing facts. “Updated weekly”, “current”, “latest version”, “must not be out of date”, “cite the source”. These point to RAG every time, and fine-tuning is the distractor.
- Behaviour words. “Consistent tone”, “house style”, “follow a strict format”, “domain terminology”, “a smaller model must match a larger one on this task”. These point to fine-tuning, or to distillation when the goal is a cheaper model.
- Cost and speed words. “Quickly”, “without training”, “minimal effort”, “no labelled data”. Prompting or RAG, never fine-tuning.
- Data-volume words. “Thousands of approved examples” supports fine-tuning. “A handful of documents” or “no labelled examples” does not.
- Wrong tool traps. Retraining to fix hallucinations about private data; RAG to change writing style; a larger prompt to “teach” the model permanently. Each treats the wrong symptom.
- Retrieval versus model. If a RAG answer is wrong, the fault is usually retrieval (chunking, search settings, missing documents), so the fix is not to fine-tune the model.
Practice questions
Question 1. A support team runs an assistant over product manuals. The manuals are revised every month, and users must be able to see which manual section an answer came from. The team has no labelled training data. What should they implement?
- A. Fine-tune the model on the manuals each month
- B. Add few-shot examples of good answers to the system prompt
- C. Retrieve passages from an index of the manuals and include them in the prompt
- D. Lower the temperature setting
Show answer
Answer: C
Content that changes monthly and a need to show sources point to retrieval-augmented generation: update the index when manuals change and return the retrieved passages as citations. Fine-tuning would need retraining after every revision, cannot cite sources and needs labelled data the team does not have. Few-shot examples help format, not knowledge of the manuals. Changing the temperature does not add missing facts.
Question 2. A company has 4,000 approved customer emails that show its exact tone and structure. An assistant built with a well-written prompt still produces replies that drift from that style, and no company facts are involved. What is the most appropriate next step?
- A. Build a RAG pipeline over the 4,000 emails and retrieve the closest one for each request
- B. Fine-tune the model on the approved emails
- C. Train a new foundation model from scratch
- D. Increase the number of retrieved chunks per request
Show answer
Answer: B
The problem is behaviour, not knowledge, the prompt has already been tried, and thousands of good examples exist, so supervised fine-tuning on those emails is the right next step. RAG supplies facts and does not reliably change style. Adding more documents to an index does not teach tone. Training a model from scratch is far more costly than the requirement justifies.
Go deeper
- Fine-tuning in Microsoft Foundry (Microsoft Learn)
- Prompt engineering techniques (Microsoft Learn)
- Customize a model in Amazon Bedrock (AWS documentation)
- Next on this site: RAG explained, AI agents explained and How to read a skills outline.