AI-901 lesson 3: how generative models work

A generative language model works by predicting the next token over and over: it reads the tokens of a prompt, scores every possible next token, picks one, adds it to the text and repeats until the response is complete. Tokens are chunks of text, often a whole short word, a part of a longer word or a punctuation mark, and each model can handle only a fixed number of them per request, called its context window. Almost every concept in this lesson follows from those two facts.

Tokens and the context window

A model does not see words or letters; it sees tokens. A common word may be one token, while a rare or long word is split into several pieces. Because of this, costs and limits are counted in tokens, not words or characters.

The context window is the maximum number of tokens the model can consider in one request. It has to hold the system prompt, the conversation so far, any supplied documents and the response itself. If the input is too long, the request fails or older material must be removed. A larger window allows longer documents and conversations, but it does not make the model smarter, and the limit differs from model to model.

Training versus inference

Two phases are easy to blur.

• Training is when the model learns patterns from very large amounts of data. It is done once (and sometimes refined later) by the model’s creator, and it fixes what the model “knows” up to a knowledge cutoff.
• Inference is using the finished model to answer a request. Everything an app does, including every prompt sent from the Foundry portal, is inference. Prompting does not retrain the model.

Two consequences matter for the exam. A model cannot know facts that appeared after its training, so current or private information must be supplied in the prompt. And a model can produce confident text that is wrong, because it predicts plausible text rather than looking facts up.

From prompt to response

The figure above shows the path. The system prompt sets standing behaviour such as role, tone and limits. The user prompt is the request of the moment. Grounding data is optional text from your own sources placed into the request so the answer can rely on it. All of this becomes tokens, the deployed model generates new tokens under the configured parameters, and those tokens are turned back into the response, often called the completion.

Parameters that shape the output

Parameters are settings sent with a request, not changes to the model.

Temperature and top-p both influence randomness, so the usual advice is to adjust one and leave the other alone. Some reasoning models do not accept these settings, so a model’s documentation is worth checking. Even at the lowest temperature, identical output is not guaranteed in every case.

Choosing a model and a deployment

Choose a model by capability and constraint, then the cheapest option that meets them.

• A large language model suits open-ended reasoning and long, complex tasks.
• A smaller model suits simple, high-volume tasks where cost and speed matter most.
• A multimodal model accepts more than text, such as images or audio, as input.
• An image-generation model creates pictures from text and is the wrong choice for chat.

In Microsoft Foundry a model becomes usable once it is deployed, and the deployment type decides where data is processed and how it is billed. At fundamentals level, know the pattern: Global Standard is the usual starting point, pay per token with the broadest model availability; Data Zone types keep processing within the US, EU or Asia Pacific zone; Standard keeps processing in the chosen Azure geography; Provisioned types reserve capacity for steady, predictable throughput; Batch types handle large asynchronous jobs at lower cost. Availability varies by model.

Worked example: a 60-person legal practice picks settings

A 60-person law firm builds an assistant that answers questions about its own policy documents. Staff need consistent, cautious answers, so the team uses a low temperature. Policy documents are long, so the team checks that the context window covers the system prompt, the supplied extract and the answer. A smaller model would be cheaper, but the questions need careful reading, so a more capable one is chosen. The firm must keep client data within the EU, which points toward a Data Zone deployment type rather than a global one. If the same firm also wanted to summarise ten thousand archived memos overnight, a batch type would suit that job better than live requests.

Easy to confuse

The topic page on prompt engineering basics goes further into writing prompts.

Key points

• Generative models predict the next token repeatedly; the context window limits how many tokens one request can hold.
• Training builds the model once; inference is every later request, and prompts do not retrain it.
• Lower temperature gives more predictable output, higher gives more varied output; top-p is a related randomness control.
• Pick the smallest model that meets the requirement; multimodal models take images or audio as input.
• The deployment type governs data processing location and billing; Global Standard is the common default.

Check yourself

This lesson sits in the first exam domain, Identify AI concepts and capabilities, which carries 40 to 45% of AI-901, and the recap page AI concepts explained summarises model choice and configuration for revision.