AI-901 lesson 10: vision and image apps

Vision in Foundry comes down to two directions. To understand a picture, an app sends it inside a prompt to a deployed multimodal model, together with a text question, and reads a text answer. To produce a picture, the app sends a text description to a deployed image-generation model and receives an image. Both are ordinary deployments in a Foundry project, so the habits from the earlier lessons apply: deploy first, test in the portal, then call it from a client.

Interpreting visual input

A multimodal chat model accepts more than text in a user message. The message holds a list of parts, and one part can be an image while another is the written question. The image is supplied either as a link to a location the service can reach or as the image data itself, encoded as text. The system prompt still sets the standing behaviour, for example “describe only what is visible and say when something is unclear.”

What the model returns is text. It can describe a scene, read visible writing, compare two photos, answer a specific question or judge whether something looks damaged. It is reasoning in language about pixels, which makes it flexible and also fallible: answers can be wrong or overconfident, so a person should review decisions that matter.

Three practical points follow. The deployed model must actually support image input; a text-only deployment will not do. Images count against the context window, so a long conversation with many photos fills it faster. And a clear question gets a better answer than “what is this?”, which is the same prompt-writing lesson as before.

Creating images with generative models

Image-generation models work the other way round. The prompt describes the subject, setting and visual style, and the model returns a new image. Microsoft’s current documentation lists the GPT-image series of models for this; the older DALL-E 3 model has been retired, so older tutorials that name it are out of date. Access to some image models can require registration, so a deployment may be unavailable until approved.

In the portal, a deployed image model can be opened in a playground, where a prompt is entered and the result appears on the page. A client app does the same through the project: it sends the prompt and options such as image size and quality to the deployment and receives the image data, which it then saves or displays. Because a generated image is created, not found, the same prompt can give different pictures, and safety filters can refuse a prompt or filter the output.

Worked example: a hardware shop’s catalogue tool

A 40-person hardware retailer wants two features. First, staff photograph returned items and the app should state whether the packaging looks damaged and mention any visible serial number. Second, the marketing person wants a few illustrated banner backgrounds for a spring sale.

The first feature is interpretation: the photo and a question go to a deployed multimodal model, and a written assessment comes back. The second is creation: a text description goes to an image-generation model. If the retailer instead needed exact serial numbers, invoice totals or table values pulled into fields, the better tool would be Content Understanding, covered in the next lesson, because that is structured extraction rather than open description.

The lightweight vision app has a simple shape: connect to the project, choose the multimodal deployment, build a user message holding the photo and the question, send it and display the text reply.

Easy to confuse

The usual trap is a question that mentions both an image and a model. Ask which direction the image travels: into the model means interpretation, out of the model means generation.

Key points

• Interpreting an image means passing it in a user message to a multimodal model, along with a text question.
• The reply to an image question is text, and it should be checked when decisions depend on it.
• Image-generation models turn a written description into a new image; the model can be tried in a playground first.
• A vision app uses the same project, deployment and client pattern as a chat app.
• The deployment must support image input, and images consume part of the context window.
• Structured fields from scans or forms point to Content Understanding rather than open-ended description.

Check yourself

This lesson sits in the Implement AI solutions by using Microsoft Foundry domain, worth 55-60% of AI-901. Revisit the domain on the Foundry implementation page, and see how responsible AI applies to generated content for the safety side.