Databricks DE Professional: developing code
Developing Code for Data Processing using Python and SQL is worth 22% of the Databricks Data Engineer Professional exam — around 13 of the 59 scored questions and the largest section. It has no counterpart on the Associate, and it is the clearest marker of the step up.
The exam is multiple choice, so you read code rather than write it. But the fluency required is real: you will be shown a snippet and asked what it produces, what is wrong with it, or which of four versions is correct.
PySpark and SQL
Both, interchangeably:
- DataFrame operations — select, filter, join, groupBy, aggregations.
- Window functions — ranking, running totals, lag and lead. Common in real pipelines and common on the exam.
- The same logic in SQL, and knowing that the two are largely equivalent in capability.
- When SQL is clearer and when the DataFrame API gives you more control.
Lazy evaluation
Fundamental, and frequently tested indirectly.
Transformations — select, filter, join — build a plan and do nothing. Actions — count, collect, write, show — trigger execution. Nothing happens until an action runs.
Consequences the exam probes:
- A chain of transformations costs nothing until it is acted on.
- Calling an action repeatedly re-executes the whole chain unless you cache.
- An error in a transformation surfaces when the action runs, not where the code is written.
Efficient transformation patterns
The recurring theme: let Spark do the work, distributed, and avoid anything that forces data through a single point or processes rows individually.
| Anti-pattern | Why it hurts | Better |
|---|---|---|
| Python UDFs for simple logic | Per-row serialisation between JVM and Python | Built-in Spark SQL functions |
collect() on a large DataFrame | Pulls everything to the driver | Aggregate in the cluster |
| Row-by-row loops | Defeats parallelism entirely | Vectorised DataFrame operations |
| Repeated actions on the same chain | Recomputes each time | Cache, if reused enough to justify it |
| Wide transformations without thought | Expensive shuffles | Filter early, broadcast small tables |
UDFs are the single most tested anti-pattern. If a question shows a Python UDF doing something a built-in function could do, replacing it is almost certainly the answer.
Joins
- Broadcast join — the small side is sent to every executor, removing the shuffle of the large side. The right answer whenever one table is small and the other is huge.
- Shuffle joins for two large tables, and why they cost what they do.
- Filtering before joining rather than after.
Writing maintainable code
- Modular, reusable transformation functions rather than one long notebook cell.
- Code that can be tested against known inputs.
- Handling errors inside a transformation rather than letting a whole job die on one bad record.
Sample questions
Question 1. A developer writes a chain of five filters and selects on a large DataFrame, then calls count() three times in sequence. What happens?
- A. The full chain executes three times, once per action, unless the result is cached
- B. The chain executes once when written, and the counts read a stored result
- C. The chain executes once and subsequent counts are automatically reused
- D. The code fails because too many filters are chained
Show answer
Answer: A
Lazy evaluation means the transformation chain is re-executed on each action, so three counts trigger the work three times unless the result is cached. The chain does not execute when written, results are not automatically reused, and the number of filters is not the issue.
Question 2. Which change is most likely to improve a job that joins a very large table to a very small lookup table?
- A. Repartition both tables on the join key
- B. Rewrite the join logic as a Python UDF
- C. Broadcast the small lookup table
- D. Increase the driver memory
Show answer
Answer: C
Broadcasting the small table removes the need to shuffle the large one, which is usually the dominant cost in such a join. Repartitioning both adds shuffle work, converting to a UDF is slower, and increasing the driver’s memory does not affect a distributed join.
Question 3. A transformation calls collect() on a 500 GB DataFrame to compute a total, then writes the result. What is wrong?
- A. The write should happen before the aggregation
- B. collect() pulls the entire dataset to the driver; the aggregation should run distributed
- C. The output format is inappropriate for aggregates
- D. The DataFrame has too few partitions
Show answer
Answer: B
collect() pulls all data to the driver, which cannot hold 500 GB and defeats distributed processing; the aggregation should be performed in the cluster. The issue is not the write, not the file format, and not partition count.
What to practise
Take a transformation you have written and rewrite it three ways: with a Python UDF, with built-in functions, and with an unnecessary collect(). Time all three on a dataset large enough to matter.
The differences are dramatic, and they convert this section from memorised rules into intuition — which is what the exam actually tests.