Databricks Data Engineer: platform and ingestion

Databricks Intelligence Platform (6%) and Data Ingestion and Loading (21%) are 27% of the Data Engineer Associate exam between them — around 12 of the 45 scored questions. The platform section is tiny; ingestion is one of the two largest on the exam.

The platform section (6%)

About three questions. Cover it in an evening:

• The lakehouse idea: one storage layer serving both analytics and data engineering, rather than a warehouse and a lake kept in sync.
• Delta Lake as the default table format — ACID transactions, schema enforcement, and time travel over files in cloud storage.
• Workspace organisation: notebooks, repos, clusters, catalogs.
• Compute basics: which compute type fits which workload, when it starts and stops, and why leaving it running costs money (details below).

That is genuinely enough for a 6% section. Do not spend a week here.

Compute types and access modes

The second platform objective asks you to know the compute services, their limits and cost models, and to pick one per workload. Current Databricks documentation and the workspace UI say compute where older material says clusters: the sidebar tab is Compute and the button is Create compute. The main options:

Instance pools keep pre-configured instances ready, which shortens start-up time for classic compute that is created often.

Access modes have new names. Classic all-purpose and job compute have an access mode that decides who can attach:

• Standard, formerly called shared: many users run code on the same compute, isolated from each other by Lakeguard. Databricks recommends it for most workloads.
• Dedicated, formerly called single user: assigned to one user or one group. The simple compute form picks Dedicated automatically for machine learning runtimes and for runtimes below 14.3.
• No isolation shared and credential passthrough are legacy modes that do not support Unity Catalog.

Expect a scenario that names a workload and asks for the compute. Interactive development by a team points to all-purpose compute in standard mode; a nightly production task points to job or serverless compute rather than an always-on all-purpose cluster; BI queries point to a SQL warehouse. Pitfall: an exam option or older study guide may still say “shared access mode” or “single user cluster”. Translate them to standard and dedicated before you answer; they are the same settings, not extra choices. The Microsoft Learn compute overview lists the current types.

Ingestion and loading (21%)

The real content. Getting data in, reliably and without waste.

Incremental over full reload

The most examined idea in the section. Reprocessing an entire source every run is the naive approach, and it is wrong for cost, runtime and correctness reasons. Incremental ingestion processes only what is new.

Any scenario describing rising cost or runtime as data accumulates is an incremental-loading question. The wrong answers will offer a bigger cluster.

Batch and streaming

• Scheduled batch loads for data arriving periodically.
• Streaming or continuous ingestion where records arrive constantly.
• The middle ground: frequent micro-batches, which most pipelines actually use.

Match the pattern to how data arrives and how fresh it must be, not to which sounds more sophisticated.

Schema handling

• Schema enforcement — Delta rejects writes that do not match, which is a feature rather than an obstacle.
• Schema evolution — allowing new columns to be added deliberately.
• What happens when a source changes shape unexpectedly, and why silently accepting it is dangerous.

Auto Loader and choosing an ingestion method

The ingestion section names Auto Loader in two objectives: use it with schema enforcement and evolution to land files in Unity Catalog tables, and choose between it, Lakeflow Connect standard and managed connectors, partner connectors and other methods. It is the Databricks answer to “new files keep arriving in cloud storage and each must be loaded once”.

Auto Loader is a Structured Streaming source called cloudFiles. It records which files it has already processed in the stream’s checkpoint, so a rerun or restart loads only new files. Run it with an availableNow trigger and it processes everything that has arrived, then stops; that is how it runs as a scheduled batch job without an always-on stream. It finds new files in one of two modes: directory listing, the default, which lists the input path, and file notification, which reads file events from a queue and is what Databricks recommends for most workloads.

Schema handling is where the questions sit:

• Setting cloudFiles.schemaLocation turns on schema inference and evolution and stores the schema there.
• In the default mode, addNewColumns, a new column makes the stream fail once after it has been added to the schema; the restart picks it up. Run it as a job so the restart is automatic.
• rescue never changes the schema and never fails; unexpected values go to the _rescued_data column.
• failOnNewColumns stops until you change the schema or remove the file. When you supply a schema yourself, the default is none, which ignores new columns.

Choosing a method is about the source, not the volume alone:

Pitfall: a stream that fails the first time a new column appears looks like a bug, and the wrong answers offer to switch off schema checks. It is the default addNewColumns behaviour; the fix is an automatic restart, or rescue if the stream must never stop. The full option list, code and file-detection detail are in the Auto Loader guide.

File formats and layout

• Columnar formats for analytical reads; row formats for record-at-a-time access.
• Compression and its trade-offs.
• The small file problem — many tiny files make reads slow because of per-file overhead. It starts at ingestion and it is worth preventing rather than fixing later.

Landing data properly

The bronze layer convention: land raw data with minimal transformation, so you can always reprocess from source. Transformation happens downstream.

Sample questions

What to practise

Set up ingestion from a storage location into a bronze Delta table, then add a file with an extra column and watch schema enforcement reject it. Enable evolution and try again. Then generate a thousand tiny files and observe read performance.

Those three experiments cover most of what this section asks, and they take an hour.