Databricks Data Engineer: Lakeflow Jobs and CI/CD
Working with Lakeflow Jobs (16%) and Implementing CI/CD (10%) are 26% of the Databricks Data Engineer Associate exam between them — around 12 of the 45 scored questions. Both are about making pipelines run reliably without someone watching.
Lakeflow Jobs (16%)
Databricks’ orchestration layer. The exam expects operational familiarity.
Jobs and tasks. A job contains one or more tasks. Tasks can run notebooks, scripts or queries, and the value of splitting work into tasks is that each can succeed, fail and retry independently.
Dependencies. Tasks declare what must complete before they run. A task depending on two upstream tasks runs only when both succeed; if an upstream task fails, dependents do not run. Any scenario describing required ordering is a dependency question, and the wrong answers offer separate jobs scheduled a few minutes apart.
Scheduling and triggers. Time-based schedules, and triggering on file arrival or another event.
Failure handling:
- Retries for transient failures — but retries only help if the task is idempotent.
- Notifications when something fails.
- Understanding partial failure: which tasks ran, which did not, and what state that leaves data in.
Parameters passed into tasks, so the same code runs against different environments or dates.
Monitoring runs — reading run history, finding which task failed and why.
Implementing CI/CD (10%)
About five questions, and the concepts are standard software practice applied to data.
Git integration. Notebooks and code in repos rather than living only in a workspace. Branching, and reviewing changes before they reach production.
Environment promotion. Development, staging and production as separate environments, with the same code promoted between them rather than rebuilt in each.
Configuration, not code. Anything environment-specific — a path, a catalog name, a schedule — belongs in a parameter or variable, not hard-coded. This is the single most testable idea in the section, and the same principle appears on nearly every platform’s certification.
Automated testing of pipelines. Testing transformations against known inputs, and validating data quality expectations as part of deployment rather than discovering problems in production.
How the two sections connect
A well-built pipeline is orchestrated and deployable: tasks with proper dependencies and retries, code in Git, configuration parameterised, and promotion automated. Questions sometimes span both, describing a team that has one and not the other.
Sample questions
Question 1. A pipeline has an extract task, two independent transform tasks, and a load task that must run only after both transforms complete. What is the correct implementation?
- A. Four separate jobs, each scheduled ten minutes apart
- B. One job with task dependencies, load depending on both transform tasks
- C. A single task running all four steps sequentially
- D. Running the tasks manually each morning in the correct order
Show answer
Answer: B
A single job with task dependencies expresses that the load depends on both transforms and prevents it running if either fails. Separate scheduled jobs cannot guarantee ordering, one combined task loses independent retry, and manual sequencing removes the point of orchestration.
Question 2. The same pipeline code must run in development and production against different catalogs. What is the appropriate approach?
- A. Maintain two copies of the notebook, one per environment
- B. Hard-code production values and edit after each deployment
- C. Parameterise the catalog name and supply it per environment
- D. Use one shared catalog for both environments
Show answer
Answer: C
Parameterising the catalog name lets one code artefact run correctly in each environment, which is the standard promotion pattern. Maintaining two copies causes drift, editing after each deployment is error-prone manual work, and a single shared catalog removes environment separation entirely.
Question 3. A task fails intermittently due to a transient connection error. Automatic retries are enabled, but reruns have started producing duplicate records. What is the underlying problem?
- A. The task is not idempotent, so retrying duplicates data
- B. The retry count is set too high
- C. The cluster is too small for the workload
- D. Retries should be disabled entirely
Show answer
Answer: A
Retries are safe only when the task is idempotent, so a task that appends rather than merges will duplicate data each time it reruns. The retry count is not the issue, the cluster is not the cause, and disabling retries leaves the pipeline failing on transient errors.
What to practise
Build a job with three tasks and a real dependency, give one task a deliberate failure, and watch what runs and what does not. Enable retries on a non-idempotent task and observe the duplicates. Then put the code in a repo and parameterise one environment-specific value.
That covers both sections, and the retry-plus-duplicates experiment connects them to the transformation section in a way the exam clearly likes.