Databricks DE Professional: cost and performance
Cost & Performance Optimisation is Section 5 of the Databricks Data Engineer Professional exam and worth 15% in the outline that applies from 9 October 2026: around 9 of the 60 scored questions and the second largest section. It grew from 13% and took over the Liquid Clustering versus partitioning comparison that used to sit under data modelling. It has no equivalent at Associate level, where tuning is folded into a general troubleshooting section.
This is the section that most distinguishes a professional data engineer from a competent one: not whether the pipeline runs, but what it costs and how long it takes.
Reading the Spark UI
The diagnostic foundation. You should be able to look at a job and identify:
- Stages and tasks — how work was divided.
- Task duration distribution — are tasks even, or are a few far slower?
- Shuffle read and write volume — how much data moved between stages.
- Spill — data written to disk because it did not fit in memory, a reliable sign of trouble.
Questions describe these symptoms in words rather than showing screenshots, so learn what each pattern means.
The four classic problems
| Problem | Symptom | Remedy |
|---|---|---|
| Data skew | A few tasks take far longer; most finish fast | Redistribute the skewed key; salt if necessary |
| Small files | Reads slow; per-file overhead dominates | Compact into fewer, larger files |
| Excessive shuffle | Large shuffle volume between stages | Broadcast small tables; filter before joining |
| Over-partitioning | Huge numbers of tiny partitions | Partition on lower-cardinality columns |
Skew versus small files is the distinction to get right. Skew concentrates in a few slow tasks while others finish. Small files slow reads broadly and evenly. The symptoms differ, and so do the fixes.
Partitioning and file layout
- Choosing partition columns by how data is queried, not by what is convenient.
- Avoiding high-cardinality partition columns, which create thousands of tiny directories.
- Compaction, and why frequent incremental writes make it necessary.
- Data layout techniques that co-locate related data so queries read less.
Choosing a Delta optimisation technique
Section 5 asks you to pick the right Delta optimisation (deletion vectors, Liquid Clustering, CLUSTER BY AUTO) for a table’s access pattern, to compare Liquid Clustering with partitioning and Z-ordering, and to explain how Unity Catalog managed tables and Predictive Optimization cut maintenance work. The questions describe a workload and offer three or four layouts.
| Technique | What it does | Pick it when |
|---|---|---|
| Deletion vectors | DELETE, UPDATE and MERGE mark changed rows in a small side file instead of rewriting each affected Parquet file; the rewrite happens later | Frequent small updates or deletes scattered across large files |
Liquid Clustering (CLUSTER BY (cols)) | Clusters data on chosen keys incrementally during OPTIMIZE; keys can be changed later without rewriting existing data | Filters on one to a few columns, including high-cardinality ones, or access patterns that may shift |
CLUSTER BY AUTO | Databricks chooses and adjusts the clustering keys from the table’s query history | Unity Catalog managed table with Predictive Optimization, and you do not want to own the key choice |
| Partitioning | Physical directories per value | Very large tables with a low-cardinality column every query filters on; Databricks advises against partitioning small tables |
| Z-ordering | OPTIMIZE ... ZORDER BY co-locates values, rewriting the data each time | Existing tables not yet moved to clustering |
Databricks recommends Liquid Clustering for new tables. It replaces partitioning and Z-ordering on a table rather than combining with them, so a question offering “partition by date and cluster by customer” on the same table is offering an invalid design.
Predictive Optimization runs maintenance such as OPTIMIZE and VACUUM automatically on Unity Catalog managed tables, so nobody has to schedule compaction jobs. That is the outline’s point about managed tables reducing operational overhead: the platform owns the layout work.
The pitfall: reading deletion vectors as a privacy feature. A row deleted with deletion vectors enabled is still physically present in the old data file until the files are rewritten and the old versions vacuumed. For a right-to-erasure request that matters, and Section 6 tests it from the other side.
For the transaction log, OPTIMIZE and VACUUM, see Delta Lake explained.
Question 1. A 40 TB Unity Catalog managed table is filtered mostly by customer_id today, but analysts expect the dominant filter to change next year. The team wants good file skipping without picking or maintaining layout keys. Predictive Optimization is enabled. Which design fits best?
- A. Partition the table by customer_id
- B. Enable automatic Liquid Clustering with CLUSTER BY AUTO
- C. Schedule a nightly OPTIMIZE ZORDER BY customer_id
- D. Enable deletion vectors on the table
Show answer
Answer: B
CLUSTER BY AUTO lets Databricks choose and adjust clustering keys from query history on a managed table with Predictive Optimization, which removes the key choice and adapts as filters change. Partitioning on customer_id creates far too many small partitions, a nightly Z-order rewrites data and fixes the key, and deletion vectors speed up row-level changes rather than file skipping.
Want more questions like this? Full Data Engineer Pro practice tests →
Caching
Useful and frequently misused:
- Cache when a DataFrame is reused several times and recomputation is expensive.
- Do not cache something read once — you pay memory for nothing.
- Cached data competes with execution memory, and over-caching causes spill.
- The Delta cache (disk cache) is a different mechanism: it keeps local copies of remote data files on the workers’ disks, so repeated reads of the same data skip cloud storage. It does not use the memory that
.cache()does.
Change Data Feed
The outline lists Change Data Feed (CDF) under optimisation because it makes downstream processing incremental. With delta.enableChangeDataFeed set on a table, consumers read only the row-level inserts, updates and deletes since a given version, with a change-type column, instead of rescanning or diffing the whole table. A plain streaming read of a Delta table expects appends; CDF is how updates and deletes reach the next layer.
Cluster cost
- Right-sizing rather than defaulting to large.
- Autoscaling for variable workloads.
- Terminating idle clusters — the single largest avoidable cost in most workspaces.
- Choosing instance types appropriate to the work; GPUs only where they earn their keep.
- Recognising that the cheapest fix is often structural, not more hardware.
That last point is the exam’s recurring instinct. When “increase the cluster size” appears alongside a structural remedy, the structural remedy is almost always correct — a bigger cluster does the same wasteful work faster and more expensively.
Sample questions
Question 2. In one stage, 3 tasks out of 200 run for 40 minutes while the rest complete in under a minute. What does this indicate?
- A. Data skew on the partitioning or join key
- B. The small file problem
- C. Insufficient driver memory
- D. Too few partitions overall
Show answer
Answer: A
A small number of very slow tasks alongside many fast ones is the signature of data skew, where certain keys hold disproportionate data. Small files slow reads broadly and evenly, insufficient driver memory affects collection to the driver, and too few partitions would slow all tasks rather than three.
Want more questions like this? Full Data Engineer Pro practice tests →
Question 3. A streaming pipeline writes micro-batches every minute. After six months, queries against the target table have become very slow. What is the most likely cause and remedy?
- A. Data skew; redistribute the partition key
- B. Insufficient cluster size; scale up the query cluster
- C. Accumulated small files; compact them into fewer, larger files
- D. Excessive retention; delete older data
Show answer
Answer: C
Frequent micro-batch writes accumulate into an enormous number of small files, and compaction into fewer larger files restores read performance. Skew would show as uneven task duration, a larger cluster pays more for the same overhead, and reducing retention discards data rather than fixing layout.
Want more questions like this? Full Data Engineer Pro practice tests →
Question 4. An engineer caches every DataFrame in a long pipeline, and the job starts spilling to disk and running slower than before. What went wrong?
- A. Caching requires an explicit refresh that was not configured
- B. Over-caching consumed memory needed for execution, causing spill
- C. Caching only works on tables, not DataFrames
- D. The cached data expired mid-run
Show answer
Answer: B
Cached data competes with execution memory, so caching indiscriminately leaves too little for processing and forces spill. Caching is not always beneficial, the cluster is not necessarily undersized, and the issue is memory pressure rather than cache expiry.
Want more questions like this? Full Data Engineer Pro practice tests →
What to practise
Build a table with a deliberately skewed key and watch the task duration distribution in the Spark UI. Then create the small file problem by writing in a tight loop, and time a query before and after compaction.
Seeing both patterns once makes them unmistakable in a written scenario — which is exactly how the exam presents them.