Databricks DE Professional: cost and performance
Cost & Performance Optimisation is worth 13% of the Databricks Data Engineer Professional exam — around 8 of the 59 scored questions and the second largest section. It has no equivalent at Associate level, where tuning is folded into a general troubleshooting section.
This is the section that most distinguishes a professional data engineer from a competent one: not whether the pipeline runs, but what it costs and how long it takes.
Reading the Spark UI
The diagnostic foundation. You should be able to look at a job and identify:
- Stages and tasks — how work was divided.
- Task duration distribution — are tasks even, or are a few far slower?
- Shuffle read and write volume — how much data moved between stages.
- Spill — data written to disk because it did not fit in memory, a reliable sign of trouble.
Questions describe these symptoms in words rather than showing screenshots, so learn what each pattern means.
The four classic problems
| Problem | Symptom | Remedy |
|---|---|---|
| Data skew | A few tasks take far longer; most finish fast | Redistribute the skewed key; salt if necessary |
| Small files | Reads slow; per-file overhead dominates | Compact into fewer, larger files |
| Excessive shuffle | Large shuffle volume between stages | Broadcast small tables; filter before joining |
| Over-partitioning | Huge numbers of tiny partitions | Partition on lower-cardinality columns |
Skew versus small files is the distinction to get right. Skew concentrates in a few slow tasks while others finish. Small files slow reads broadly and evenly. The symptoms differ, and so do the fixes.
Partitioning and file layout
- Choosing partition columns by how data is queried, not by what is convenient.
- Avoiding high-cardinality partition columns, which create thousands of tiny directories.
- Compaction, and why frequent incremental writes make it necessary.
- Data layout techniques that co-locate related data so queries read less.
Caching
Useful and frequently misused:
- Cache when a DataFrame is reused several times and recomputation is expensive.
- Do not cache something read once — you pay memory for nothing.
- Cached data competes with execution memory, and over-caching causes spill.
Cluster cost
- Right-sizing rather than defaulting to large.
- Autoscaling for variable workloads.
- Terminating idle clusters — the single largest avoidable cost in most workspaces.
- Choosing instance types appropriate to the work; GPUs only where they earn their keep.
- Recognising that the cheapest fix is often structural, not more hardware.
That last point is the exam’s recurring instinct. When “increase the cluster size” appears alongside a structural remedy, the structural remedy is almost always correct — a bigger cluster does the same wasteful work faster and more expensively.
Sample questions
Question 1. In one stage, 3 tasks out of 200 run for 40 minutes while the rest complete in under a minute. What does this indicate?
- A. Data skew on the partitioning or join key
- B. The small file problem
- C. Insufficient driver memory
- D. Too few partitions overall
Show answer
Answer: A
A small number of very slow tasks alongside many fast ones is the signature of data skew, where certain keys hold disproportionate data. Small files slow reads broadly and evenly, insufficient driver memory affects collection to the driver, and too few partitions would slow all tasks rather than three.
Question 2. A streaming pipeline writes micro-batches every minute. After six months, queries against the target table have become very slow. What is the most likely cause and remedy?
- A. Data skew; redistribute the partition key
- B. Insufficient cluster size; scale up the query cluster
- C. Accumulated small files; compact them into fewer, larger files
- D. Excessive retention; delete older data
Show answer
Answer: C
Frequent micro-batch writes accumulate into an enormous number of small files, and compaction into fewer larger files restores read performance. Skew would show as uneven task duration, a larger cluster pays more for the same overhead, and reducing retention discards data rather than fixing layout.
Question 3. An engineer caches every DataFrame in a long pipeline, and the job starts spilling to disk and running slower than before. What went wrong?
- A. Caching requires an explicit refresh that was not configured
- B. Over-caching consumed memory needed for execution, causing spill
- C. Caching only works on tables, not DataFrames
- D. The cached data expired mid-run
Show answer
Answer: B
Cached data competes with execution memory, so caching indiscriminately leaves too little for processing and forces spill. Caching is not always beneficial, the cluster is not necessarily undersized, and the issue is memory pressure rather than cache expiry.
What to practise
Build a table with a deliberately skewed key and watch the task duration distribution in the Spark UI. Then create the small file problem by writing in a tight loop, and time a query before and after compaction.
Seeing both patterns once makes them unmistakable in a written scenario — which is exactly how the exam presents them.