AI-901 lesson 7: system and user prompts

A system prompt holds the standing instructions that apply to every turn of a conversation, such as persona, tone, scope and refusals. A user prompt is the message for one turn, such as a question or a task. A third role, assistant, carries the model’s earlier replies. A chat model receives all three as one list of messages, so the quality of the first two decides most of what comes back, and an exam question that asks where an “always” or “never” rule belongs is asking about the system prompt.

The roles and what belongs where

The test for where a sentence goes is simple: would it still be true in the next conversation? If yes, it is a system prompt rule. Safety-relevant restrictions are best backed by platform guardrails, not by wording alone.

Building an effective prompt

Good prompts remove guesswork. Five ingredients cover most needs:

1. Role. Who the model is acting as: “a support assistant for a travel agency”.
2. Task. What to do, stated plainly and first, because instructions near the start and end tend to carry more weight.
3. Format. What the output should look like: three bullets, a table, a fixed set of labels, or a specified structure that another program will read.
4. Constraints. Length, language, topics to avoid, and an “out” such as “say you do not know when the answer is not in the provided text”. Giving the model an out reduces invented answers.
5. Examples. One or more sample inputs with the desired outputs. Few-shot examples guide only the current request and do not retrain the model, and a prompt with none is called zero-shot.

Clear separators between instructions and pasted material, such as headings or delimiters, help the model tell them apart. Sampling settings such as temperature are separate from the prompt text. The prompt engineering basics page extends these techniques.

Grounding with your own data

A model knows only what it learned in training, which has a cut-off and contains nothing about a company’s private documents. Left alone it may answer confidently and wrongly. Grounding fixes this by supplying the relevant facts at question time. The common pattern is retrieval-augmented generation (RAG), in four steps: the question is used to search a knowledge source, the best passages are retrieved, those passages are added to the prompt as context, and the model writes an answer based on them. The system prompt typically instructs it to answer only from the supplied text and to cite it.

Grounding differs from fine-tuning. Fine-tuning changes the model’s weights using extra training. Grounding leaves the model unchanged and changes only the input, so updated documents are used immediately. The topic pages on RAG and fine-tuning vs RAG vs prompting cover the trade-offs.

Worked example: a policy assistant that drifts

A 200-person firm deploys an assistant for its HR policies. Testers report that it switches language, chats about holiday destinations, and quotes a leave rule that does not exist.

The fixes map to the ingredients. The first two are standing behaviours, so they go in the system prompt: reply in English unless asked otherwise, discuss only HR policy, and politely decline other topics. The invented rule is a grounding problem. The policy documents are retrieved and added as context, and the system prompt adds the out: answer only from the provided policy text, and say so when the answer is not there. Leaving such rules to individual user prompts would be fragile.

Easy to confuse

• System prompt vs user prompt. Standing rules versus a single request.
• Prompt vs parameters. Wording shapes content, while temperature and similar settings shape how varied the sampling is.
• Grounding vs fine-tuning. Supplying facts at question time versus changing the model through extra training.

Common mistakes follow from these: vague tasks (“make it better”), conflicting instructions, burying the format request in a long paragraph, assuming the model remembers earlier conversations, and trusting an ungrounded answer without checking it.

Key points

• The system prompt sets standing behaviour for every turn, and the user prompt carries one request.
• Strong prompts state role, task, format, constraints and, where useful, examples.
• Giving the model permission to say it does not know reduces invented answers.
• RAG retrieves relevant passages and adds them to the prompt so the answer is grounded.
• Grounding changes the input, fine-tuning changes the model.

Check yourself

This lesson sits in the implementation domain (55-60% of AI-901), after the Foundry basics and before agents. The domain recap is at Foundry implementation explained.