Databricks DE Professional: monitoring and deploying
Monitoring and Alerting (10%) and Debugging and Deploying (10%) are 20% of the Databricks Data Engineer Professional exam between them — around 12 of the 59 scored questions. Both are about what happens after a pipeline is written.
Monitoring and alerting (10%)
What to instrument. Three layers, and questions test whether you know they are different:
| Layer | Tells you |
|---|---|
| Infrastructure | Is the cluster healthy? |
| Pipeline | Did the job run, and how long did it take? |
| Data | Is the output actually correct? |
The failure the exam returns to: a job that succeeds while producing wrong data. Infrastructure and pipeline monitoring both report green. Only data-level monitoring catches it. If a scenario describes a metric being wrong for weeks while everything “worked”, the gap is data quality monitoring.
Meaningful thresholds. Alerting on everything produces noise nobody reads. Alert on conditions that require action: a job that failed, a runtime that doubled, a row count that collapsed, a quality expectation breached.
Trends over snapshots. A job taking 40 minutes is not informative. A job that took 12 minutes last month and 40 today is.
Freshness. Knowing when a table was last successfully updated, and alerting when it goes stale — a pipeline that silently stops running is worse than one that fails loudly.
Debugging and deploying (10%)
Diagnosing failures:
- Reading job run history to find which task failed and why.
- Distinguishing transient failures — a network blip, a throttled API — from systematic ones. Retries help the first and merely delay the second.
- Using the Spark UI to find where a job is actually spending time or failing.
- Reproducing a failure: which data, which parameters, which version of the code.
Deploying:
- Promoting the same artefact between environments rather than rebuilding.
- Configuration parameterised, never hard-coded.
- Deployment patterns that limit blast radius, and the ability to roll back.
- Testing pipelines before production: transformations against known inputs, and quality expectations as a gate.
The connection between them: a pipeline you cannot debug is one you deployed without thinking about observability. Questions sometimes span both sections, describing a team that ships quickly and cannot diagnose anything.
Sample questions
Question 1. A pipeline's runtime has crept from 15 minutes to 90 minutes over four months. No alert fired at any point. What monitoring approach would have surfaced this?
- A. Alerting only when the job fails
- B. Collecting cluster CPU metrics
- C. Alerting on runtime deviation from an established baseline
- D. A daily email listing all job completions
Show answer
Answer: C
Alerting on a trend or a deviation from a baseline catches gradual degradation, which a fixed failure-only alert never will. Alerting solely on failure misses a job that still succeeds, infrastructure metrics do not capture runtime creep meaningfully, and a daily summary nobody acts on is not an alert.
Question 2. A task fails roughly once a week with a connection timeout to an external API, and succeeds when rerun manually. What is the appropriate response?
- A. Configure retries with backoff, ensuring the task is idempotent
- B. Rewrite the pipeline to avoid the external API
- C. Increase the cluster size
- D. Continue rerunning manually when it fails
Show answer
Answer: A
An intermittent timeout that succeeds on rerun is a transient failure, and configured retries with backoff handle it automatically provided the task is idempotent. Rewriting to avoid the API changes the requirement, a larger cluster does not affect an external timeout, and manual reruns do not scale.
Question 3. A new pipeline version is deployed straight to production and corrupts a downstream table. Recovery takes a day. Which two practices would most have limited the damage?
- A. More verbose logging and a larger cluster
- B. Deploying at night and notifying users in advance
- C. A longer retention period on the source data
- D. Testing transformations against known inputs, plus a deployment pattern with fast rollback
Show answer
Answer: D
Testing against known inputs before production and a deployment pattern allowing rapid rollback together limit both the likelihood and the duration of the damage. More logging aids diagnosis after the fact, a larger cluster is irrelevant, and deploying at night only changes who notices first.
What to practise
Take a pipeline you own and answer three questions: how would you know if it stopped running, how would you know if it ran but produced wrong data, and how long would rolling back the last change take?
If any answer is “someone would eventually notice”, you have found what this section is testing. Then add one freshness alert and one quality expectation, and time a rollback.