DP-750 pipelines and workloads explained
Deploy and maintain data pipelines and workloads is worth 30–35% of DP-750, tied with data preparation as the heaviest domain. It covers designing pipelines, scheduling them with Lakeflow Jobs, moving code through environments with Git and bundles, and then keeping everything fast, cheap and observable. It is the production half of the role, and candidates who only develop in notebooks find it the hardest.
Design and implement data pipelines
You are asked to decide the order of operations (ingest, validate, transform, publish) and to choose between a notebook pipeline and Lakeflow Spark Declarative Pipelines. Declarative pipelines suit table-to-table transformations where you describe the result and let the engine work out dependencies, orchestration and incremental processing, with expectations for quality. Notebooks suit custom logic, calls to external systems or steps that do not map to tables. Many designs combine both inside one job.
Error handling applies at every level: try/except blocks and explicit failure in notebooks, retry settings on tasks, expectation actions in pipelines, and dependencies that stop downstream tasks when something upstream fails. A notebook pipeline uses task dependencies to express precedence: task B runs only if task A succeeds.
Implement Lakeflow Jobs
A job is a set of tasks with dependencies. Know how to:
- create a job and set its compute, parameters and permissions;
- configure triggers: scheduled, file arrival, table update, or continuous;
- schedule with a cron expression and a time zone;
- add alerts for start, success, failure or duration;
- set automatic restarts (task retries) for transient failures, in both jobs and pipelines.
The troubleshooting actions are separate and the exam tests the difference: repair reruns only failed tasks and their dependants, restart starts the run or pipeline again, stop cancels it, and run starts a new one.
Implement development lifecycle processes
Git
Use Git folders to connect the workspace to a repository. Best practice is feature branches, pull requests with review, and resolving conflicts before merging to main. Production code deploys from the main branch, not from someone’s workspace.
Testing
| Test | Checks |
|---|---|
| Unit | One function or transformation on small data |
| Integration | Components working together, such as ingest into a table |
| End-to-end | The whole pipeline from source to published output |
| UAT | Business users confirm the output is what they need |
Bundles
A bundle describes jobs, pipelines and other resources as code in YAML, with targets such as dev and prod. Microsoft’s outline calls them Declarative Automation Bundles; older Databricks material says Asset Bundles. You configure and package a bundle, validate it, and deploy with the Databricks CLI (databricks bundle deploy -t <target>) or through REST APIs. The CLI is the usual answer in CI/CD questions.
Monitor, troubleshoot and optimise
Cost and performance
Monitor cluster consumption to find over-sized or idle compute; fix it with autoscaling, shorter termination, or serverless and job compute. For slow Spark work, use the Spark UI, the DAG, and the query profile:
| Symptom | Likely cause | Typical fix |
|---|---|---|
| One task much slower than others | Data skew | Salting, adaptive query execution, repartition by a better key |
| Disk spill in stages | Not enough memory per task | Larger nodes, fewer partitions per core |
| Huge shuffle read and write | Wide joins or aggregations | Broadcast the small table, filter earlier |
| Repeated slow scans | Missing caching or poor layout | Cache, cluster the table |
Delta maintenance
OPTIMIZE compacts many small files into larger ones to speed up reads. VACUUM deletes files that are no longer referenced and are older than the retention period (7 days by default), which reduces storage cost but limits time travel.
Azure Monitor
Stream Databricks logs to a Log Analytics workspace, query them with KQL, and create Azure Monitor alerts for failures, long runs or resource thresholds. The outline treats Azure Monitor as part of operating Databricks on Azure.
Sample questions
Question 1. A transformation is a chain of table-to-table steps with data quality rules, and the team wants dependencies and incremental processing handled by the platform instead of hand-written orchestration. What should they use?
- A. One large notebook with manual ordering
- B. Lakeflow Spark Declarative Pipelines
- C. An all-purpose cluster running ad hoc commands
- D. A cron script on a virtual machine
Show answer
Answer: B
Lakeflow Spark Declarative Pipelines let you declare tables and expectations and handle dependency ordering and incremental processing. A single large notebook needs manual orchestration. An all-purpose cluster is compute, not a pipeline approach. A scheduled script on a VM is outside the platform and adds operational work.
Want more questions like this? Full DP-750 practice tests →
Question 2. A CI/CD workflow must deploy a job and pipeline defined as code to the dev target of a bundle. Which command is appropriate?
- A. git clone on the workspace
- B. Running each notebook manually
- C. OPTIMIZE on the target tables
- D. databricks bundle deploy -t dev
Show answer
Answer: D
The Databricks CLI deploys a bundle to a named target with databricks bundle deploy -t dev, usually after validate. Git clone only copies source files. Running the notebook manually does not create or update the job definitions. OPTIMIZE is a Delta table maintenance command and has no role in deployment.
Want more questions like this? Full DP-750 practice tests →
Question 3. Storage costs are rising because a heavily updated Delta table keeps many data files that no query references any more, all older than the retention period. Which command removes them?
- A. VACUUM
- B. OPTIMIZE
- C. ANALYZE TABLE
- D. DROP TABLE
Show answer
Answer: A
VACUUM deletes files no longer referenced by the table and older than the retention threshold. OPTIMIZE compacts small files but leaves old versions in place. ANALYZE TABLE computes statistics. Dropping the table would delete current data as well.
Want more questions like this? Full DP-750 practice tests →
What to practise
Build a job with a failing middle task and use repair. Package it in a bundle with dev and prod targets and deploy with the CLI. Cause skew with a join on a lopsided key and find it in the Spark UI. Run OPTIMIZE and VACUUM and check the table history. Connect a workspace to Log Analytics and create one alert. Then read the exam tips.