Databricks Data Engineer: platform and ingestion
Databricks Intelligence Platform (6%) and Data Ingestion and Loading (21%) are 27% of the Data Engineer Associate exam between them — around 12 of the 45 scored questions. The platform section is tiny; ingestion is one of the two largest on the exam.
The platform section (6%)
About three questions. Cover it in an evening:
- The lakehouse idea: one storage layer serving both analytics and data engineering, rather than a warehouse and a lake kept in sync.
- Delta Lake as the default table format — ACID transactions, schema enforcement, and time travel over files in cloud storage.
- Workspace organisation: notebooks, repos, clusters, catalogs.
- Compute basics: clusters, when they start and stop, and why leaving them running costs money.
That is genuinely enough for a 6% section. Do not spend a week here.
Ingestion and loading (21%)
The real content. Getting data in, reliably and without waste.
Incremental over full reload
The most examined idea in the section. Reprocessing an entire source every run is the naive approach, and it is wrong for cost, runtime and correctness reasons. Incremental ingestion processes only what is new.
Any scenario describing rising cost or runtime as data accumulates is an incremental-loading question. The wrong answers will offer a bigger cluster.
Batch and streaming
- Scheduled batch loads for data arriving periodically.
- Streaming or continuous ingestion where records arrive constantly.
- The middle ground: frequent micro-batches, which most pipelines actually use.
Match the pattern to how data arrives and how fresh it must be, not to which sounds more sophisticated.
Schema handling
- Schema enforcement — Delta rejects writes that do not match, which is a feature rather than an obstacle.
- Schema evolution — allowing new columns to be added deliberately.
- What happens when a source changes shape unexpectedly, and why silently accepting it is dangerous.
File formats and layout
- Columnar formats for analytical reads; row formats for record-at-a-time access.
- Compression and its trade-offs.
- The small file problem — many tiny files make reads slow because of per-file overhead. It starts at ingestion and it is worth preventing rather than fixing later.
Landing data properly
The bronze layer convention: land raw data with minimal transformation, so you can always reprocess from source. Transformation happens downstream.
Sample questions
Question 1. A source system delivers a few hundred new files daily into cloud storage. The current job reads the entire location each night and runtime has grown from minutes to hours. What is the correct fix?
- A. Switch to incremental ingestion processing only new files
- B. Provision a larger cluster for the nightly job
- C. Delete source files after each successful run
- D. Reduce the job frequency to weekly
Show answer
Answer: A
Processing only newly arrived files addresses the cause, which is repeatedly reading data that has already been ingested. A larger cluster makes wasteful work faster at higher cost, deleting source files loses the ability to reprocess, and reducing frequency increases latency without improving efficiency.
Question 2. An upstream system adds a new column to its export without notice. The ingestion job fails. What does this indicate about the target table?
- A. The table is corrupted and must be rebuilt
- B. The cluster is too small to handle the extra column
- C. Schema enforcement rejected the write, which is the intended protection
- D. The job lacks permission to write to the table
Show answer
Answer: C
A write rejected because the incoming data does not match the table’s schema is schema enforcement working as designed, protecting the table from unexpected changes. It is not corruption, not a cluster sizing issue, and not a permissions problem.
Question 3. Which describes the appropriate content of a bronze layer table?
- A. Aggregated metrics ready for business reporting
- B. Raw ingested data with minimal transformation, retained so downstream layers can be rebuilt
- C. Fully cleansed and conformed data ready for analysts
- D. Temporary scratch data deleted after each run
Show answer
Answer: B
The bronze layer holds raw ingested data with minimal transformation so it can always be reprocessed from source, with cleansing and modelling happening downstream. Aggregated business metrics belong in gold, fully cleansed conformed data in silver, and a bronze table is not a temporary scratch area.
What to practise
Set up ingestion from a storage location into a bronze Delta table, then add a file with an extra column and watch schema enforcement reject it. Enable evolution and try again. Then generate a thousand tiny files and observe read performance.
Those three experiments cover most of what this section asks, and they take an hour.