Lakeflow Declarative Pipelines (ex-DLT)

Lakeflow Declarative Pipelines is the Databricks framework, formerly called Delta Live Tables (DLT), in which you declare the tables you want and the platform works out the order, retries and incremental processing. The decision rule for exams: if a scenario is a chain of table-to-table transformations with data quality rules and the team does not want to hand-write orchestration or checkpoint handling, the answer is a declarative pipeline. If the work is arbitrary code, external calls or steps that do not produce tables, the answer is a notebook or task in Lakeflow Jobs.

One feature, five names

The same product appears under several names, and a question can use any of them. Databricks documentation now says “Lakeflow pipelines” for the product and “Apache Spark Declarative Pipelines” (SDP) for the open-source framework it extends. The Python module was dlt and is now pyspark.pipelines, imported as dp; the old dlt module still works, but Databricks recommends the new one.

Treat all of these as the same thing unless the question is about the open-source difference. AUTO CDC, expectations and the queryable event log exist only on the Databricks side, not in open-source SDP.

The three dataset types

A pipeline produces datasets of three kinds, and choosing between them is the most tested skill.

Both streaming tables and materialized views are Unity Catalog managed tables, so other jobs and queries can read them. The rule: append-only source and low latency, streaming table; aggregation or changing source, materialized view. A streaming table expects an append-only source and processes each record once, which is why aggregations and joins over changing data belong in materialized views.

A small pipeline with a quality rule

This SQL pipeline lands JSON files with Auto Loader, drops bad rows, and aggregates:

CREATE OR REFRESH STREAMING TABLE orders_raw
AS SELECT * FROM STREAM read_files('/Volumes/main/sales/landing/orders/', format => 'json');

CREATE OR REFRESH STREAMING TABLE orders_clean(
  CONSTRAINT valid_amount EXPECT (amount > 0) ON VIOLATION DROP ROW
) AS SELECT * FROM STREAM(orders_raw);

CREATE OR REFRESH MATERIALIZED VIEW daily_revenue
AS SELECT order_date, SUM(amount) AS revenue FROM orders_clean GROUP BY order_date;

Nothing says which table runs first. The pipeline reads the dependencies and runs orders_raw, then orders_clean, then daily_revenue. In Python the equivalent uses @dp.table for a streaming read and @dp.materialized_view for a batch read. Dataset functions must only define and return a DataFrame, so calls such as collect(), count() or saveAsTable() do not belong in them.

Expectations: warn, drop or fail

An expectation is a SQL boolean rule attached to a dataset, with a name. The action decides what happens to rows that break it.

Quarantining invalid rows into a separate table is a pattern built from these tools, not a fourth action. Constraints cannot call custom Python functions, external services or subqueries on other tables. Expectations are not supported with AUTO CDC FROM SNAPSHOT.

Change data capture with AUTO CDC

AUTO CDC applies a feed of inserts, updates and deletes to a target streaming table, handling out-of-order events through a sequencing column. stored_as_scd_type = 1 keeps only the latest row; 2 keeps history with __START_AT and __END_AT columns. It replaced APPLY CHANGES, which has the same syntax and still works. Use AUTO CDC FROM SNAPSHOT when the source offers only periodic snapshots and no change feed.

Triggered or continuous, and where it runs

Triggered mode refreshes everything available and stops; continuous mode keeps tables fresh as data arrives. Triggered suits updates every ten minutes or slower, continuous suits latency between about ten seconds and a few minutes, at the cost of an always-running cluster. Mode is independent of table type. Databricks recommends serverless for new pipelines; serverless pipelines require Unity Catalog. Updates started from the UI use fast-start behavior with retries off for quick debugging, while updates started from a job or the API use automatic retries.

Which exams test this

The Data Engineer Professional outline changed on October 9, 2026, so older study notes use the previous section split.

How the questions are built

Questions give a requirement and offer three Databricks tools that all “work”. The constraint words decide: “incrementally”, “append-only” and “exactly once” point to a streaming table; “aggregate”, “join” and “source rows are updated” point to a materialized view; “must not load invalid rows” points to drop, and “stop the pipeline” or “unacceptable” points to fail. “Out-of-order change events” and “history of changes” point to AUTO CDC with SCD type 2. A frequent trap is offering a view when other jobs need to query the result, because views are not materialized. Another is choosing continuous mode for a daily refresh, which wastes cost. The related ideas live in Auto Loader, Delta Lake and batch vs streaming; the Bronze, Silver, Gold layering is in medallion architecture.

Go deeper

• Lakeflow pipelines concepts (Microsoft Learn)
• Manage data quality with pipeline expectations
• The AUTO CDC APIs
• What happened to @dlt? (Python reference)
• Related on this site: Unity Catalog, the Data Engineer Associate study plan and how to study for certification exams