Databricks Data Engineer Professional test
Twenty questions across the eight sections of the Databricks Data Engineer Professional exam, weighted roughly as the real exam is. This is a significant step up from the associate exam: expect production judgement rather than syntax recall.
Developing Code for Data Processing using Python and SQL
Question 1. A streaming job must guarantee that each source record affects the target exactly once, even after a driver restart. What makes this possible?
- A. Increasing the retry count
- B. Checkpointed offsets combined with an idempotent or transactional sink
- C. Writing to a temporary table and renaming it
- D. Disabling automatic restart
Show answer
Answer: B
Checkpointing offsets together with an idempotent or transactional sink gives exactly-once effect across restarts. Retries alone can duplicate writes.
Question 2. A Python UDF processes 400 million rows and dominates job runtime. What is the first change to consider?
- A. Replace it with built-in SQL expressions or a vectorised UDF
- B. Double the cluster size
- C. Increase the driver memory only
- D. Cache the input DataFrame
Show answer
Answer: A
Replacing a row-at-a-time Python UDF with built-in expressions, or a vectorised UDF, removes serialisation overhead per row. Scaling the cluster pays for the same inefficiency.
Question 3. A join between a 4 TB fact table and a 30 MB dimension spills heavily. What should be verified first?
- A. Whether the fact table is sorted
- B. Whether the dimension has a primary key constraint
- C. Whether the small side is being broadcast, and why the optimiser chose otherwise
- D. Whether the cluster has enough disk
Show answer
Answer: C
A dimension that small should be broadcast; if it is not, the threshold or statistics are wrong. Adding partitions manages the shuffle rather than removing it.
Question 4. Code must behave identically whether run on a batch backfill or an incremental stream. Which approach supports this best?
- A. Maintain two separate implementations
- B. Run the batch version and copy results into the stream target
- C. Disable the streaming version during backfill
- D. One shared transformation, parameterised by read and write mode
Show answer
Answer: D
Expressing the transformation once and varying only the read and write modes keeps one code path, so backfill and incremental results agree by construction.
Question 5. A widely reused transformation must be testable outside a notebook. What is the appropriate structure?
- A. Extract it into functions in a version-controlled module with unit tests
- B. Copy the cells into each notebook that needs it
- C. Store the SQL in a table and read it at runtime
- D. Document the steps in a wiki page
Show answer
Answer: A
Extracting the logic into functions in a versioned module allows unit testing and reuse across jobs. Notebook-only logic cannot be tested in isolation.
Cost & Performance Optimisation
Question 6. A daily job's cost has tripled while data volume grew 20%. Profiling shows heavy file listing. What is the likely cause?
- A. The cluster is undersized
- B. Small file accumulation in the source or target tables
- C. Too few columns are selected
- D. The job runs too early in the day
Show answer
Answer: B
An accumulation of small files makes listing and opening dominate runtime, which scales far worse than data volume. Compaction addresses it directly.
Question 7. A table is almost always filtered on two columns with high cardinality. Which physical design helps most?
- A. Partition by both columns
- B. Sort the data alphabetically by the first column only
- C. Cluster or z-order the table on those columns
- D. Convert the table to CSV
Show answer
Answer: C
Clustering or z-ordering on those columns co-locates related values so file skipping is effective. Partitioning on high-cardinality columns creates too many tiny partitions.
Question 8. Interactive development clusters are left running overnight. Which control reduces cost with least disruption?
- A. Auto-termination after a short idle period
- B. Removing cluster creation permissions from engineers
- C. Reducing the number of workers to one permanently
- D. Restricting development to office hours by policy only
Show answer
Answer: A
Auto-termination after idle stops unused clusters without affecting active work. Removing cluster creation rights blocks legitimate work.
Data Transformation, Cleansing, and Quality
Question 9. Downstream consumers must never see rows that fail validation, but the rows must remain investigable. What design fits?
- A. Drop failing rows silently
- B. Load everything and let consumers filter
- C. Fail the whole job on any invalid row
- D. Route failing rows to a quarantine table and publish only valid rows
Show answer
Answer: D
Writing failing rows to a quarantine table while only valid rows reach the consumer table preserves evidence without contaminating downstream data.
Question 10. A quality rule must fail the pipeline when more than 1% of rows are invalid, but tolerate less. What is required?
- A. A row-level constraint that rejects each invalid row
- B. A threshold expectation evaluated across the batch
- C. A manual review after each run
- D. A warning comment in the code
Show answer
Answer: B
A threshold-based expectation evaluated against the batch enforces the tolerance. Row-level failure alone cannot express a proportion.
Monitoring and Alerting
Question 11. A streaming job silently falls behind during peak hours. Which metric most directly reveals this?
- A. Input backlog or source lag over time
- B. Notebook cell execution count
- C. Number of columns written
- D. Workspace user count
Show answer
Answer: A
Growing input backlog or consumer lag shows the job is not keeping pace. Cluster CPU may look healthy while the backlog grows.
Question 12. An alert should fire only when a failure is genuinely actionable, not for transient retried errors. How should it be configured?
- A. Alert on every task error immediately
- B. Alert only once per month in a summary
- C. Alert on final failure after retries are exhausted
- D. Disable alerting and review logs weekly
Show answer
Answer: C
Alerting on final failure after retries are exhausted, with a duration or count condition, suppresses noise while catching real problems.
Ensuring Data Security and Compliance
Question 13. Personal data must be removable on request, but the table is append-only with historical versions retained. What must be considered?
- A. Nothing; deleting the current row is sufficient
- B. Retained historical versions and downstream copies must also be addressed within the retention policy
- C. The table must be converted to CSV first
- D. Only the dashboard needs updating
Show answer
Answer: B
Deletion must propagate through retained versions and downstream copies, so retention settings and vacuum policy must align with the erasure obligation.
Question 14. A job's service principal has workspace administrator rights for convenience. What is the finding?
- A. No issue, as jobs are trusted
- B. It increases cost
- C. It slows the job down
- D. Excessive privilege; the principal should be scoped to only what the job needs
Show answer
Answer: D
The principal holds far more privilege than the job requires, so any compromise or bug has maximum blast radius. Least privilege is the correction.
Debugging and Deploying
Question 15. A job fails in production but succeeds in development with the same code. What should be compared first?
- A. Environment configuration, input data characteristics and permissions
- B. The notebook’s formatting
- C. The names of the developers
- D. The workspace colour theme
Show answer
Answer: A
Environment configuration, data volume and permissions are the usual differences between environments. The code is already established as identical.
Question 16. A deployment must be reversible within minutes if data quality degrades. What supports this?
- A. A detailed runbook with no automation
- B. Keeping the previous notebook open in a browser tab
- C. Versioned deployment plus table versioning and a tested rollback procedure
- D. Deploying only on Fridays
Show answer
Answer: C
Versioned code deployment combined with table versioning and a tested rollback procedure allows both logic and data state to be restored.
Data Ingestion & Acquisition
Question 17. A source system provides a change data capture feed with inserts, updates and deletes. What must the target logic handle?
- A. Append every event and let consumers work it out
- B. Ordered merge semantics including deletes and late or out-of-order events
- C. Inserts only, ignoring updates and deletes
- D. A full reload each night regardless of the feed
Show answer
Answer: B
Applying ordered changes with merge semantics, including deletes and out-of-order events, is required for correctness. Appending all events produces a log, not current state.
Question 18. An upstream API rate-limits aggressively and occasionally returns partial pages. What should ingestion include?
- A. Backoff with resumable pagination and completeness validation
- B. A tighter retry loop with no delay
- C. Ignoring partial pages as they are rare
- D. Increasing cluster size to fetch faster
Show answer
Answer: A
Backoff with resumable pagination and validation of completeness protects against both throttling and silent truncation.
Data Governance
Question 19. An analyst asks which downstream tables would be affected if a silver table's schema changes. What answers this reliably?
- A. A search through notebook source
- B. The job scheduler’s task list
- C. The cluster event log
- D. Table lineage in the governance catalog
Show answer
Answer: D
Lineage recorded by the governance catalog maps downstream dependencies. Searching code manually misses dynamic references and external consumers.
Question 20. Definitions of a key business metric differ between two gold tables. What is the governance remedy?
- A. Document both definitions and let consumers choose
- B. Establish one certified, owned definition and converge both tables on it
- C. Delete one of the tables without consultation
- D. Rename the columns to avoid confusion
Show answer
Answer: B
A single certified definition with ownership, published and referenced by both consumers, removes the divergence. Documenting both preserves the problem.
How did you do?
Sixteen or more correct suggests you are close. Below fourteen, the section guides here are the fastest route back. Databricks does not publish a passing score for this exam, so be sceptical of specific numbers quoted elsewhere.