Delta Lake explained: log, ACID, time travel

Delta Lake is an open table format that puts a transaction log on top of ordinary Parquet files in cloud storage, which gives a data lake the behaviour of a database table: atomic writes, consistent reads, schema checks and history. The decision rule for exams: if a scenario needs updates, deletes, merges, safe concurrent writers, or the ability to read an earlier state of the data, the answer is a Delta table, not plain Parquet, CSV or JSON files.

Layer one: files that never change

A Delta table is a folder. Inside are Parquet data files and a subfolder called _delta_log. Data files are immutable: Delta never edits a Parquet file in place. An UPDATE writes new files containing the changed rows and marks the old files as removed. Every behaviour below follows from that one design choice.

Layer two: the transaction log

The log is an ordered series of small commit files, one per successful write. Each commit lists actions such as “add this data file” and “remove that data file”. The current state of the table is whatever you get by replaying the log from the start (the log is periodically summarised into checkpoint files so readers do not replay everything). The format is public: the Delta transaction log protocol is an open specification, which is why several engines can read the same table.

A write is not visible until its commit file lands in the log. That gives you:

• Atomicity. A job that fails halfway leaves orphan data files that no commit references, so readers never see half a write.
• Consistency and isolation. A reader works from one snapshot, the table as of a log version, even while writers keep committing.
• Concurrency by optimism. Writers do not lock. Before committing, Delta checks whether another commit since the snapshot conflicts with the staged change. If it does, the write fails with a concurrent-modification exception rather than corrupting the table (Delta documentation, concurrency control). Appends from two jobs usually coexist; two jobs rewriting the same files do not.
• Durability. The commit is stored in the same cloud storage as the data.

Together that is ACID. Schema enforcement is part of the same story: a write whose columns do not match the table schema is rejected unless schema evolution is deliberately enabled.

A worked example: four commits

An orders table receives these operations in order:

Version 3 reads only files C and D. Files A and B still exist on storage, which is why this works:

SELECT * FROM orders VERSION AS OF 1;
SELECT * FROM orders TIMESTAMP AS OF '2026-10-01';
DESCRIBE HISTORY orders;

The table describes the classic copy-on-write behaviour. With deletion vectors enabled (an option on Databricks whose default for new tables varies by workspace and region), an UPDATE or DELETE can mark rows as removed in a small side file instead of rewriting the whole data file; old versions stay readable in the same way.

Version 1 resolves to files A and B. That is time travel: reading an old snapshot costs nothing extra, because the old files were never deleted. DESCRIBE HISTORY lists the versions, and RESTORE rolls a table back to one.

OPTIMIZE and VACUUM do opposite jobs

Because every small write adds files, tables drift into thousands of tiny files and reads slow down.

Bin-packing OPTIMIZE is idempotent: a second run on the same data has no effect, according to Databricks documentation. On Databricks, predictive optimization can run OPTIMIZE and VACUUM automatically for Unity Catalog managed tables; otherwise you schedule it. Note the two separate retention settings: history in the log is kept for 30 days by default (delta.logRetentionDuration), while deleted data files must age 7 days (delta.deletedFileRetentionDuration) before VACUUM may remove them. Time travel reaches back only as far as both allow, so vacuuming with a short retention breaks it. Databricks recommends keeping at least 7 days, partly because a shorter window can delete files that a long-running job still needs.

Which exams test this

How exam questions are built around Delta

The stem usually describes a symptom and asks for the one feature that fixes it. Read the symptom first:

• “Many small files, slow queries” points to OPTIMIZE. “Storage cost keeps growing” points to VACUUM. Candidates swap them because both touch files.
• “Recover the table as it was before a bad update” points to time travel or RESTORE, and the follow-up trap is a VACUUM that already ran.
• “Two jobs write at once and one fails” is expected behaviour from optimistic concurrency, not a bug.
• “Source added a column and the write failed” is schema enforcement working. The fix is deliberate schema evolution, not disabling checks.
• “Apply inserts, updates and deletes from a feed in one statement” is MERGE.

Constraint words that decide it: recover, audit a previous state (time travel), reduce small files (OPTIMIZE), reclaim storage (VACUUM), without copying data (the open format and shared log).

Practice questions

Go deeper

• What is Delta Lake in Databricks?: transaction log, ACID and the default format.
• Delta Lake: remove unused data files with VACUUM and OPTIMIZE: retention thresholds and compaction.
• Delta transaction log protocol: the open specification.
• Related topic: DAX basics for the reporting layer on top, and How to study for certification exams for planning your revision.