MLA-C02 model and FM development
ML Model and Foundation Model (FM) Development is worth 24% of MLA-C02. AWS frames it as selecting modelling approaches, training models, tuning hyperparameters, analysing performance and managing model versions — now covering both traditional ML and foundation models.
Choosing an approach
Given a business problem, pick the modelling approach:
- Classification for categories, regression for continuous values, clustering for unlabelled grouping, forecasting for time series.
- Built-in algorithms versus your own code versus a pre-trained service.
- And the question C02 adds: traditional ML or a foundation model?
That last decision has a practical answer. A well-defined, high-volume, narrow task — classifying transactions, predicting a number — is usually cheaper and more reliable with a traditional model. An open-ended language or multimodal task is foundation model territory.
AWS AI services versus a custom model
Two objectives in task 2.1 sit together here. Objective 2.1.8 asks you to apply AWS AI services to business problems, naming Textract, Rekognition, Comprehend and Transcribe. Objective 2.1.4 asks you to weigh custom solutions against managed services, pre-trained models and foundation models. On this exam you are not just recognising a service as on AIF-C01; you decide whether a pre-trained API is enough or whether the requirement forces you into SageMaker AI.
| Service | Business problem it solves | Built-in way to adapt it |
|---|---|---|
| Amazon Textract | Text, forms and tables from scanned documents; dedicated analysis for invoices, receipts and identity documents | Queries to ask for specific fields |
| Amazon Rekognition | Objects, scenes, faces, text and unsafe content in images and video | Custom Labels for your own object classes |
| Amazon Comprehend | Entities, key phrases, language, sentiment and PII in text | Custom classification and custom entity recognition |
| Amazon Transcribe | Speech to text, streaming or from files in S3 | Custom vocabularies; PII redaction |
The order of preference follows cost and effort. First, the pre-trained service as it is. Second, the service’s own customisation (Custom Labels, a custom Comprehend classifier), which still needs only labelled examples, not an ML pipeline. Third, a foundation model in Amazon Bedrock when the output is open-ended. Last, a model you build, train and host in SageMaker AI, which buys full control over the algorithm, features and hosting at the price of owning all of it. A scenario that mentions a proprietary algorithm, features that come from your own tabular data, or a requirement no service covers justifies that last step; a scenario that only says “our documents are domain-specific” usually does not.
Exam pitfall: “train a model in SageMaker AI to detect sentiment in support tickets” looks like the engineering answer on an engineering exam, but Comprehend already does it, and custom classification covers your own categories. Choose the custom build only when the scenario names a constraint the managed service cannot meet. Also check availability: some features, such as Comprehend topic modeling, are closed to new customers.
The full input-to-output cheat sheet and the list of closed services are in the AWS AI services guide.
Training and hyperparameter tuning
- Hyperparameters are the settings you choose before training: learning rate, batch size, tree depth, number of epochs. Distinguish them from parameters, which the model learns.
- Automatic tuning searches the space for you, and the exam expects you to know it exists and roughly what it costs.
- Managing training cost: spot capacity, right-sized instances, early stopping.
Analysing model performance
The most testable area in the domain, because metric choice is where engineers go wrong.
| Metric | Use when |
|---|---|
| Accuracy | Classes are balanced |
| Precision | False positives are expensive |
| Recall | False negatives are expensive |
| F1 | You need a balance of the two |
| AUC-ROC | Comparing classifiers across thresholds |
| RMSE / MAE | Regression |
The classic scenario: a heavily imbalanced dataset where accuracy looks superb because the model predicts the majority class every time. Fraud, disease, defects, churn — any rare positive — is a precision and recall question, never an accuracy question.
Also know overfitting and underfitting by their signatures: overfitting is strong on training data and weak on new data; underfitting is weak on both.
Evaluating generated text and RAG
Precision and recall need a single right answer, which generated text rarely has. Task 2.3 of the outline therefore adds a separate toolkit for foundation models: NLP metrics such as BLEU, ROUGE, BERTScore and semantic similarity (2.3.8), LLM-as-a-judge and content quality checks (2.3.9), human evaluation (2.3.7) and RAG retrieval accuracy (2.3.10).
| Method | What it compares | Typical fit |
|---|---|---|
| BLEU | n-gram overlap with a reference, precision-oriented | Translation |
| ROUGE | n-gram overlap with a reference, recall-oriented | Summarisation |
| BERTScore | Token embeddings against a reference, so paraphrases still score | Free-form answers with a reference |
| Semantic similarity | Distance between embeddings of output and reference | Meaning-level match |
| LLM-as-a-judge | A second model scores outputs against a rubric | Open-ended output with no reference |
| Human evaluation | People rate or compare outputs | Tone, helpfulness, final sign-off |
The first four all need a reference answer. When you have none, you judge the output instead: a judge model at scale, humans where nuance or accountability matters.
On AWS, Amazon Bedrock evaluations wraps these as jobs: automatic evaluation with built-in metrics on a prompt dataset, human-based evaluation with your own work team, model-as-a-judge evaluation, and evaluations for Bedrock Knowledge Bases. The knowledge base evaluations score retrieval on its own (are the retrieved chunks relevant and do they cover what is needed) and retrieval plus generation together (is the answer correct and faithful to the retrieved context). That split is the core of 2.3.10: when a RAG answer is wrong, first find out whether retrieval failed or generation did, because the fixes are different (chunking, embeddings or reranking versus prompt or model).
Keep evaluations reproducible: a fixed dataset, versioned prompts (Bedrock Prompt Management) and recorded runs (MLflow on SageMaker AI), so a score from today can be compared with next month’s (2.3.1).
Exam pitfall: a high ROUGE or BLEU score means the wording overlaps with the reference, not that the answer is true. A scenario about hallucinated facts in summaries is not solved by “track ROUGE”; it needs a faithfulness or groundedness check, by a judge model or humans, against the source. Also watch for an answer that applies reference metrics to a task where no reference answers exist.
The full method, including how judge prompts are built, is in the LLM evaluation guide.
Model versioning and experiment tracking
- Versioning models so you know exactly what is deployed.
- Recording which data, code and hyperparameters produced a given model — reproducibility is an engineering requirement, not a nicety.
- A registry as the controlled path from experiment to production.
Objective 2.3.1 names MLflow on SageMaker AI for reproducible experiments, and objective 3.3.6 in the deployment domain asks you to manage model versions for repeatability and audits with SageMaker Model Registry or MLflow on SageMaker AI. The two work at different moments:
| Need | Tool | What you get |
|---|---|---|
| Compare training attempts and reproduce one | MLflow tracking | Runs with parameters, metrics and artifacts |
| Know which version is approved and deployable | Model Registry | Model groups holding numbered model versions, each with an approval status |
In the Model Registry, a model group collects every version trained for one problem, and each registered version carries metadata such as training metrics and lineage. A new version usually starts as PendingManualApproval. Setting it to Approved can start CI/CD deployment, and in the SageMaker AI project templates moving an approved version to Rejected redeploys the latest version that is still approved, which is your rollback path. A pipeline can set the status automatically with a condition step on an evaluation metric.
The two tools connect. With Model Registry sync turned on (it is opt-in), registering a model in MLflow on SageMaker AI also creates the matching model group and model version in the Model Registry, and lifecycle stages can be set through MLflow aliases. Training metrics and lineage from the MLflow run then show up on the registered version, which is what an auditor asks for: which data, code and settings produced the model that is serving.
Exam pitfall: “which settings produced the best model” is a tracking question; “which version may go to production, who approved it, and how do we roll back” is a registry question. An answer that keeps model files in a versioned S3 bucket stores artefacts but gives no approval workflow or lineage. MLflow concepts such as aliases are covered in the MLflow basics guide.
Foundation model development
New in C02, and framed as a ladder of increasing cost:
- Prompt engineering — cheapest, fastest, try first.
- Retrieval-augmented generation — when answers must come from your content, especially content that changes.
- Fine-tuning — when you need consistent style, format or domain behaviour.
- Continued pre-training — rarely, at significant cost.
AWS rewards climbing that ladder only as far as the requirement demands. A scenario solvable with a better prompt is not a fine-tuning question.
Sample questions
Question 1. A model must identify a rare manufacturing defect occurring in 0.5% of units. Missing a defect is far more costly than inspecting a good unit unnecessarily. Which metric should be optimised?
- A. Accuracy
- B. Recall on the defect class
- C. Precision on the defect class
- D. Root mean squared error
Show answer
Answer: B
Missing a defect is a false negative, so recall on the defect class is the metric to optimise, accepting more false positives as the cheaper error. Accuracy is meaningless at this class balance, precision would minimise the less costly error, and RMSE applies to regression.
Want more questions like this? Full MLA-C02 practice tests →
Question 2. A team needs a foundation model to answer from internal policy documents that are revised monthly, and wants the cheapest approach that meets the requirement. What should they implement?
- A. Fine-tune the model each month on the revised documents
- B. Continued pre-training on the full policy corpus
- C. Retrieval-augmented generation over an index of the documents
- D. Select a foundation model with more parameters
Show answer
Answer: C
Retrieval-augmented generation retrieves the current documents at query time, keeping answers current without repeated training, which is the cheapest approach meeting a monthly-change requirement. Fine-tuning monthly is expensive and repeats forever, continued pre-training more so, and a bigger model does not supply the documents.
Want more questions like this? Full MLA-C02 practice tests →
Question 3. An engineer must be able to reproduce exactly how a model currently in production was produced, six months from now. What is required?
- A. Versioned model registry entries linked to the data version, code version and hyperparameters used
- B. Endpoint invocation logs retained for twelve months
- C. A written description of the approach in the project wiki
- D. A copy of the model artefact stored in S3
Show answer
Answer: A
Reproducibility requires recording the training data version, code version and hyperparameters alongside the registered model version, so the artefact can be traced and rebuilt. Endpoint logs record inference rather than training, a description is not a record of inputs, and a saved copy of the artefact alone does not explain how it was made.
Want more questions like this? Full MLA-C02 practice tests →
Question 4. A RAG assistant built on Amazon Bedrock Knowledge Bases sometimes gives wrong answers. The team wants to know whether the cause is the retrieved content or the generated response before changing anything. What should it do?
- A. Run Amazon Bedrock evaluations on the knowledge base, scoring retrieval separately from retrieval plus generation
- B. Track the ROUGE score of answers against reference summaries
- C. Switch to a larger foundation model and compare the answers
- D. Lower the temperature so the answers become deterministic
Show answer
Answer: A
A knowledge base evaluation can score retrieval on its own and retrieval plus generation together, which separates a retrieval failure from a generation failure. ROUGE against reference summaries measures wording overlap, not where the error comes from. A larger model or a lower temperature changes the generator without showing whether retrieval was the problem.
Want more questions like this? Full MLA-C02 practice tests →
Question 5. A logistics company wants to pull the totals, dates and line items out of thousands of scanned supplier invoices each day. It has a small ML team and wants the least operational overhead. What should the ML engineer use?
- A. Amazon Comprehend custom entity recognition on the scanned images
- B. Amazon Rekognition text detection
- C. Amazon Textract invoice analysis
- D. A custom document model trained and hosted in SageMaker AI
Show answer
Answer: C
Amazon Textract extracts text, forms and tables from scanned documents and has dedicated invoice analysis, with no model to train or host. Comprehend analyses text that has already been extracted, Rekognition is built for images and video rather than document structure, and a custom SageMaker AI model adds training and hosting work the scenario wants to avoid.
Want more questions like this? Full MLA-C02 practice tests →
What to practise
Train one model on a deliberately imbalanced dataset and look at accuracy next to precision and recall. The gap between them is the single most examined idea in this domain, and seeing it once on your own data makes those questions automatic.