MLA-C02 model and FM development
ML Model and Foundation Model (FM) Development is worth 24% of MLA-C02. AWS frames it as selecting modelling approaches, training models, tuning hyperparameters, analysing performance and managing model versions — now covering both traditional ML and foundation models.
Choosing an approach
Given a business problem, pick the modelling approach:
- Classification for categories, regression for continuous values, clustering for unlabelled grouping, forecasting for time series.
- Built-in algorithms versus your own code versus a pre-trained service.
- And the question C02 adds: traditional ML or a foundation model?
That last decision has a practical answer. A well-defined, high-volume, narrow task — classifying transactions, predicting a number — is usually cheaper and more reliable with a traditional model. An open-ended language or multimodal task is foundation model territory.
Training and hyperparameter tuning
- Hyperparameters are the settings you choose before training: learning rate, batch size, tree depth, number of epochs. Distinguish them from parameters, which the model learns.
- Automatic tuning searches the space for you, and the exam expects you to know it exists and roughly what it costs.
- Managing training cost: spot capacity, right-sized instances, early stopping.
Analysing model performance
The most testable area in the domain, because metric choice is where engineers go wrong.
| Metric | Use when |
|---|---|
| Accuracy | Classes are balanced |
| Precision | False positives are expensive |
| Recall | False negatives are expensive |
| F1 | You need a balance of the two |
| AUC-ROC | Comparing classifiers across thresholds |
| RMSE / MAE | Regression |
The classic scenario: a heavily imbalanced dataset where accuracy looks superb because the model predicts the majority class every time. Fraud, disease, defects, churn — any rare positive — is a precision and recall question, never an accuracy question.
Also know overfitting and underfitting by their signatures: overfitting is strong on training data and weak on new data; underfitting is weak on both.
Model versioning and experiment tracking
- Versioning models so you know exactly what is deployed.
- Recording which data, code and hyperparameters produced a given model — reproducibility is an engineering requirement, not a nicety.
- A registry as the controlled path from experiment to production.
Foundation model development
New in C02, and framed as a ladder of increasing cost:
- Prompt engineering — cheapest, fastest, try first.
- Retrieval-augmented generation — when answers must come from your content, especially content that changes.
- Fine-tuning — when you need consistent style, format or domain behaviour.
- Continued pre-training — rarely, at significant cost.
AWS rewards climbing that ladder only as far as the requirement demands. A scenario solvable with a better prompt is not a fine-tuning question.
Sample questions
Question 1. A model must identify a rare manufacturing defect occurring in 0.5% of units. Missing a defect is far more costly than inspecting a good unit unnecessarily. Which metric should be optimised?
- A. Accuracy
- B. Recall on the defect class
- C. Precision on the defect class
- D. Root mean squared error
Show answer
Answer: B
Missing a defect is a false negative, so recall on the defect class is the metric to optimise, accepting more false positives as the cheaper error. Accuracy is meaningless at this class balance, precision would minimise the less costly error, and RMSE applies to regression.
Question 2. A team needs a foundation model to answer from internal policy documents that are revised monthly, and wants the cheapest approach that meets the requirement. What should they implement?
- A. Fine-tune the model each month on the revised documents
- B. Continued pre-training on the full policy corpus
- C. Retrieval-augmented generation over an index of the documents
- D. Select a foundation model with more parameters
Show answer
Answer: C
Retrieval-augmented generation retrieves the current documents at query time, keeping answers current without repeated training, which is the cheapest approach meeting a monthly-change requirement. Fine-tuning monthly is expensive and repeats forever, continued pre-training more so, and a bigger model does not supply the documents.
Question 3. An engineer must be able to reproduce exactly how a model currently in production was produced, six months from now. What is required?
- A. Versioned model registry entries linked to the data version, code version and hyperparameters used
- B. Endpoint invocation logs retained for twelve months
- C. A written description of the approach in the project wiki
- D. A copy of the model artefact stored in S3
Show answer
Answer: A
Reproducibility requires recording the training data version, code version and hyperparameters alongside the registered model version, so the artefact can be traced and rebuilt. Endpoint logs record inference rather than training, a description is not a record of inputs, and a saved copy of the artefact alone does not explain how it was made.
What to practise
Train one model on a deliberately imbalanced dataset and look at accuracy next to precision and recall. The gap between them is the single most examined idea in this domain, and seeing it once on your own data makes those questions automatic.