Databricks DE Professional: developing code
Developing Code for Data Processing using Python and SQL is Section 1 of the Databricks Data Engineer Professional exam and worth 23% in the outline that applies from 9 October 2026: around 14 of the 60 scored questions and the largest section. It has no counterpart on the Associate, and it is the clearest marker of the step up.
The exam is multiple choice, so you read code rather than write it. But the fluency required is real: you will be shown a snippet and asked what it produces, what is wrong with it, or which of four versions is correct.
What Section 1 lists
The October 2026 guide spells the section out as twelve objectives. Grouped:
| Area | What the guide names |
|---|---|
| Project and dependencies | Python project structure for Declarative Automation Bundles (formerly Asset Bundles); troubleshooting PyPI, wheel and source-archive installs on serverless, pipeline and bundle-deployed compute |
| Functions | UDFs in Pandas, Python and SQL, including Unity Catalog functions |
| Pipelines | Lakeflow Declarative Pipelines with Auto Loader for streaming; streaming table versus materialised view; AUTO CDC (formerly APPLY CHANGES) with SCD Type 1 and 2 via stored_as_scd_type |
| Jobs | Lakeflow Jobs via UI, API and CLI; If/Else and For Each tasks |
| Compute | Serverless environments and performance mode, high-memory notebook tasks, disallowing retries |
| Testing | assertDataFrameEqual, assertSchemaEqual, DataFrame.transform and test frameworks |
| Streaming | Structured Streaming versus declarative pipelines; stateful processing, covered below |
The rest of this page concentrates on the code-reading skills that run through all of them.
PySpark and SQL
Both, interchangeably:
- DataFrame operations — select, filter, join, groupBy, aggregations.
- Window functions — ranking, running totals, lag and lead. Common in real pipelines and common on the exam.
- The same logic in SQL, and knowing that the two are largely equivalent in capability.
- When SQL is clearer and when the DataFrame API gives you more control.
Lazy evaluation
Fundamental, and frequently tested indirectly.
Transformations — select, filter, join — build a plan and do nothing. Actions — count, collect, write, show — trigger execution. Nothing happens until an action runs.
Consequences the exam probes:
- A chain of transformations costs nothing until it is acted on.
- Calling an action repeatedly re-executes the whole chain unless you cache.
- An error in a transformation surfaces when the action runs, not where the code is written.
Efficient transformation patterns
The recurring theme: let Spark do the work, distributed, and avoid anything that forces data through a single point or processes rows individually.
| Anti-pattern | Why it hurts | Better |
|---|---|---|
| Python UDFs for simple logic | Per-row serialisation between JVM and Python | Built-in Spark SQL functions |
collect() on a large DataFrame | Pulls everything to the driver | Aggregate in the cluster |
| Row-by-row loops | Defeats parallelism entirely | Vectorised DataFrame operations |
| Repeated actions on the same chain | Recomputes each time | Cache, if reused enough to justify it |
| Wide transformations without thought | Expensive shuffles | Filter early, broadcast small tables |
UDFs are the single most tested anti-pattern. If a question shows a Python UDF doing something a built-in function could do, replacing it is almost certainly the answer.
Joins
- Broadcast join — the small side is sent to every executor, removing the shuffle of the large side. The right answer whenever one table is small and the other is huge.
- Shuffle joins for two large tables, and why they cost what they do.
- Filtering before joining rather than after.
Writing maintainable code
- Modular, reusable transformation functions rather than one long notebook cell.
- Code that can be tested against known inputs.
- Handling errors inside a transformation rather than letting a whole job die on one bad record.
Stateful Structured Streaming
The October 2026 outline adds an objective to Section 1: apply stateful-processing semantics in Structured Streaming, namely watermarks, output modes, foreachBatch and checkpoints, so that state recovers after a failure with exactly-once results. At this level the exam does not ask what streaming is. It gives you a query and asks what it will emit, keep or repeat.
Watermarks. A windowed aggregation or a stream-stream join keeps state for every open window or unmatched key. withWatermark("event_time", "10 minutes") tells Spark how late an event may arrive and still count; state older than the watermark can be dropped. Without a watermark that state grows for as long as the query runs. The guarantee only covers data inside the threshold: an event later than that may or may not be included, so do not build logic that depends on late rows being counted.
Output modes. Append writes only rows that will never change again, so an aggregation in append mode needs a watermark and emits a window only once the watermark passes its end. Update writes rows that changed since the last trigger. Complete rewrites the whole result table every trigger, which only suits small aggregations.
foreachBatch. It hands each micro-batch to your function as an ordinary DataFrame plus a batch ID, which is how you run a MERGE into Delta or write to several targets from one stream. Databricks documents it as at-least-once: after a failure, a batch can run again. A MERGE on a key is safe to repeat; a plain append is not, unless you use the batch ID or Delta’s idempotent-write options to skip a batch already written.
Checkpoints. The checkpoint location stores source offsets and state. Each query needs its own location, and restarting from it is what makes recovery possible. Replayable source + checkpoint + idempotent sink is the exactly-once recipe. Some changes to a query, such as altering its stateful operations, are not compatible with an existing checkpoint.
The pitfall: treating the checkpoint alone as the exactly-once guarantee. If the sink inside foreachBatch is a non-idempotent append, a replayed batch duplicates rows even though the checkpoint is intact.
For the batch-versus-streaming decision and how triggers work, see batch vs streaming.
Question 1. A streaming query uses foreachBatch to MERGE each micro-batch into a Delta table by key and also append the same rows to an audit table. After a failure and restart, the audit table holds one micro-batch twice; the merged table is correct. What explains this?
- A. The checkpoint location was corrupted and must be deleted
- B. The watermark was too short, so late rows were written twice
- C. foreachBatch is at-least-once; the replayed batch was absorbed by the MERGE but not by the plain append
- D. The query should use complete output mode
Show answer
Answer: C
foreachBatch is at-least-once, so the interrupted batch was processed again on restart. The keyed MERGE is idempotent and absorbed the replay; the plain append is not, so it needs the batch ID or idempotent-write options to skip a batch already written. The checkpoint did its job, the watermark governs late data rather than replays, and complete mode would not fix an append.
Want more questions like this? Full Data Engineer Pro practice tests →
Sample questions
Question 2. A developer writes a chain of five filters and selects on a large DataFrame, then calls count() three times in sequence. What happens?
- A. The full chain executes three times, once per action, unless the result is cached
- B. The chain executes once when written, and the counts read a stored result
- C. The chain executes once and subsequent counts are automatically reused
- D. The code fails because too many filters are chained
Show answer
Answer: A
Lazy evaluation means the transformation chain is re-executed on each action, so three counts trigger the work three times unless the result is cached. The chain does not execute when written, results are not automatically reused, and the number of filters is not the issue.
Want more questions like this? Full Data Engineer Pro practice tests →
Question 3. Which change is most likely to improve a job that joins a very large table to a very small lookup table?
- A. Repartition both tables on the join key
- B. Rewrite the join logic as a Python UDF
- C. Broadcast the small lookup table
- D. Increase the driver memory
Show answer
Answer: C
Broadcasting the small table removes the need to shuffle the large one, which is usually the dominant cost in such a join. Repartitioning both adds shuffle work, converting to a UDF is slower, and increasing the driver’s memory does not affect a distributed join.
Want more questions like this? Full Data Engineer Pro practice tests →
Question 4. A transformation calls collect() on a 500 GB DataFrame to compute a total, then writes the result. What is wrong?
- A. The write should happen before the aggregation
- B. collect() pulls the entire dataset to the driver; the aggregation should run distributed
- C. The output format is inappropriate for aggregates
- D. The DataFrame has too few partitions
Show answer
Answer: B
collect() pulls all data to the driver, which cannot hold 500 GB and defeats distributed processing; the aggregation should be performed in the cluster. The issue is not the write, not the file format, and not partition count.
Want more questions like this? Full Data Engineer Pro practice tests →
What to practise
Take a transformation you have written and rewrite it three ways: with a Python UDF, with built-in functions, and with an unnecessary collect(). Time all three on a dataset large enough to matter.
The differences are dramatic, and they convert this section from memorised rules into intuition — which is what the exam actually tests.