Delta Lake, by contrast

Description

This is the only lesson, out of this ecosystem's eight guides — and out of this guide's sixty-four lessons — where you're going to find the name Delta Lake. You aren't going to install it, you aren't going to create a single Delta table, you aren't going to run a single line of code against it. You're going to do something different: look at the exact same problem lessons 3 through 6 of this module just solved on Iceberg — snapshots that accumulate cost, files that need compacting, cleanup that needs doing safely — and confirm, quoting Delta Lake's official documentation, that this format solves the same problem with a completely different metadata mechanism underneath, and near-identical time travel syntax on top.

Connection to the module. Everything you saw in this module's lessons 2 through 6 — catalogs, expire_snapshots, compaction, remove_orphan_files — has, in Delta Lake, a direct equivalent. This lesson doesn't repeat that work: it contrasts it, once, with quoted evidence.

Why the market names both in almost the same breath

The market audit that validates this whole guide — cited in the DESIGN doc — found something concrete: real job postings asking, literally, for "Apache Iceberg or Delta Lake table formats," with no distinction between the two, as if they were interchangeable alternatives for the same role. This isn't a coincidence or a slip by whoever wrote the posting — it's an accurate reflection of the 2026 market's state: two open-source table formats, both built on Parquet, both solving the same ceiling plain Parquet alone could never solve (the problem this guide's module 1 opened by denying: ACID transactions, snapshots, time travel, safe schema evolution). A data engineer who only knows one of the two, in 2026, knows half the vocabulary their own market uses interchangeably.

An analogy: two court filing systems, one same tribunal

This guide's module 2 compared Iceberg's metadata → manifest list → manifest files → data files chain to a court case's file: the cover page points at this hearing's evidence index, which points at the evidence folders, which point at the actual photos — a tree, with several levels of indirection. Delta Lake solves the same "reconstruct how the case looked at any point in the past" problem with a different file: a minute book, a strictly sequential list of entries ("entry N: exhibit X was added, exhibit Y was removed"), where reconstructing the state at any point means reading every entry from the beginning up to that point — or, to avoid reading thousands of old entries, consulting a periodic summary someone already prepared every ten entries. Both systems answer exactly the same question — "what did this look like at moment X?" — one does it by navigating a tree of references, the other by replaying a sequence of events.

The metadata mechanism: tree vs. flat log

This guide's module 2 already taught, in depth, Iceberg's tree: metadata.json (with the complete list of snapshots) points at a manifest list (Avro, one per snapshot), which points at one or more manifest files (Avro, each one enumerating data files), which finally point at the Parquet files. Every level exists so a query never has to open more than strictly necessary — module 2, lesson 7's Going Deeper section already explained why several inspection methods exist with different costs, precisely because of this tree structure.

Delta Lake solves the same problem with a deliberately simpler structure: the _delta_log, a directory with a strictly ordered sequence of JSON files, one per commit — 00000000000000000000.json, 00000000000000000001.json, 00000000000000000002.json, and so on — where each file describes, in plain JSON, which data files got added (add) and which got removed (remove) in that exact transaction. To avoid having to reconstruct the state by reading thousands of JSON files from commit zero, Delta Lake automatically generates a checkpoint: every ten commits (by default), it writes a Parquet file with the complete state consolidated up to that point, so reconstructing any recent version only requires reading the latest checkpoint plus a handful of JSON files after it.

	Apache Iceberg	Delta Lake
Metadata structure	Tree: metadata.json → manifest list → manifest files	Flat log: sequence of JSON commits (_delta_log/*.json)
Metadata file format	JSON (metadata) + Avro (manifest list, manifest files)	JSON (each commit) + Parquet (periodic checkpoints)
Every write creates	A new snapshot	A new version (same concept, different name)
Periodic summary	Doesn't apply the same way — every manifest list is already self-contained	Parquet checkpoint every 10 commits, by default

The distinction isn't cosmetic: a tree lets a query that only needs to know "what files does the current snapshot have" never have to touch the complete commit history — exactly why table.inspect.files() (this module's lesson 3) kept responding instantly, no matter how many redundant nights had accumulated. A flat log, instead, needs the checkpoint mechanism to achieve something similar — without checkpoints, reconstructing the current state of a table with thousands of commits would mean reading thousands of JSON files, one by one, in order.

Time travel: near-identical syntax, a different mechanism underneath

This is where the contrast becomes, for anyone who already knows Iceberg, surprisingly comfortable. Delta Lake's official documentation, in "Table batch reads and writes," documents exactly two forms of time travel, in SQL:

-- (representative -- Delta Lake syntax, not executed in this guide)
SELECT * FROM kiosko.dim_product VERSION AS OF 3;
SELECT * FROM kiosko.dim_product TIMESTAMP AS OF '2026-08-14 00:00:00';

And its DataFrame equivalent:

# (representative -- Delta Lake API, not executed in this guide)
df_v3 = spark.read.format("delta").option("versionAsOf", 3).load("/warehouse/kiosko/dim_product")
df_at = spark.read.format("delta").option("timestampAsOf", "2026-08-14 00:00:00").load(...)

Compare it against what you already really ran in this guide's module 3 and module 6:

# Iceberg -- PyIceberg, really executed in module 3
v1_rows = table.scan(snapshot_id=snap_v1).to_arrow()

# Iceberg -- Spark SQL, equivalent syntax (official documentation)
# SELECT * FROM local.kiosko.dim_product VERSION AS OF <snapshot-id>;
# SELECT * FROM local.kiosko.dim_product TIMESTAMP AS OF '2026-08-15 00:00:00';

Iceberg also supports, in Spark SQL, exactly the same two keywords — VERSION AS OF and TIMESTAMP AS OF — with the difference that "version" in Iceberg is the snapshot-id (a long integer, system-assigned, never predictable in advance, this guide's same hard rule repeated since module 3) and in Delta Lake it's a sequential version number, 0, 1, 2, 3, ..., one per _delta_log commit — easier to predict at a glance, but exactly as useless to hardcode in a real pipeline: the correct version to recover still depends on how many commits happened before, information only the system itself knows for certain.

Maintenance: OPTIMIZE and VACUUM, this module's same two problems

Delta Lake names its maintenance operations differently, but they precisely solve the same two problems this module's lessons 4 through 6 worked on Iceberg:

This module's problem	Iceberg operation	Delta Lake operation
Small files fragmenting the current snapshot/version (lesson 4)	rewrite_data_files (Spark, representative in this guide)	OPTIMIZE <table>
Old snapshots/versions nobody needs anymore (lesson 3)	expire_snapshots (PyIceberg, really executed in lesson 5)	VACUUM <table> [RETAIN num HOURS]
Orphan files on disk (lesson 6)	remove_orphan_files (Spark, representative in this guide)	(VACUUM itself covers this case too)

Delta Lake's VACUUM combines, in a single operation, what Iceberg splits into two — expire_snapshots (metadata) and remove_orphan_files (filesystem) — it directly deletes data files that no longer belong to any version within the retention window. Its default retention value is seven days — even more conservative than Iceberg's remove_orphan_files's default three days — with the exact same safety logic you already saw in this module's lesson 6: too short a window risks deleting files a write in progress still needs.

The 2026 convergence: "which format?" is no longer a functionality question

Up to here, this lesson's contrast could read as "two distinct formats, each with its own closed implementation." The 2026 evidence says the opposite. Databricks — the company behind Delta Lake, and the original engine behind the whole Spark ecosystem — built Delta UniForm: a feature that, on a Delta table's same Parquet files, also generates Iceberg-compatible metadata, so a client that only knows how to read Iceberg can read that same table with no conversion. And, beyond UniForm, Databricks formally proposed that Delta Lake 5.0 adopt Iceberg v4's metadata tree as its native content structure — a single on-disk structure, readable and writable by clients of both formats, with no translation layer in between. As a 2026 technical report on the state of lakehouse formats sums it up: "Read that plainly: the largest Delta stakeholder proposing that Delta's next major version converge onto Iceberg's next metadata design."

This isn't a niche detail. That same report confirms that, in 2026, both Databricks and Snowflake — the two largest commercial engines in the ecosystem, each with its own native format — "read and write [Iceberg] natively" ("Databricks and Snowflake both read and write it natively"), and concludes, on which format to choose for a new project in 2026: "Iceberg, and in 2026 this is barely a debate" — not because Delta Lake stopped working or lost real adoption, but because "the engine breadth, the neutral governance, the catalog ecosystem, and the fact that every other format now builds bridges to it make it the lowest-regret default." And this same module's lesson 2 already anticipated the piece that closes the loop: Unity Catalog, Databricks's governance catalog, natively implements Iceberg's REST Catalog protocol — the same provider that invented Delta Lake built its next-generation catalog on its historic competitor's open protocol.

The honest conclusion, and the exact reason this guide teaches Iceberg as its main vehicle without building a second, parallel Delta implementation: in 2026, the question "Iceberg or Delta Lake?" stopped being a question about what each one is capable of doing — both solve, with evidence quoted in this very lesson, the same ACID, snapshots, time travel, and maintenance problem — and became a question about which catalog and which engine your organization already has, knowing the bridges between both formats — UniForm, the proposed convergence for Delta 5.0, Unity Catalog speaking REST — keep growing every quarter.

Diagram: same problem, two paths converging

flowchart TB
    P["The same problem:\nACID, snapshots/versions,\ntime travel, maintenance"]
    P --> I["Apache Iceberg\ntree: metadata -> manifest list -> manifest files\nVERSION AS OF (snapshot-id)\nexpire_snapshots + rewrite_data_files + remove_orphan_files"]
    P --> D["Delta Lake\nflat log: _delta_log/*.json + checkpoints\nVERSION AS OF (sequential version)\nOPTIMIZE + VACUUM"]
    I -.->|"UniForm: Iceberg metadata\non Delta files"| D
    D -.->|"Proposed Delta 5.0:\nnative Iceberg v4\nmetadata tree, no translation"| I
    I -.->|"Unity Catalog:\nimplements REST Catalog\n(this module's lesson 2)"| D

Common mistakes

Thinking that, because this lesson quotes Delta Lake extensively, this guide "also teaches Delta Lake a little." What happens: someone, after reading this lesson, believes they already know how to use Delta Lake in practice, or looks for Delta code exercises in the rest of this guide. Why it happens: the amount of technical detail quoted in this lesson — exact syntax, metadata mechanisms, maintenance operations — can feel like enough to "know how to use it." How to spot it: if you look for a Delta Lake code block marked as executed (not representative) in any lesson of this guide, you're not going to find one — not in this lesson, not in any other. How to fix it: this lesson gives you the vocabulary and the conceptual map to recognize Delta Lake when you run into it in the market — a job posting, an existing project, a technical discussion — not the practice of building with it. Learning to really operate it, with executed code, is the job of a source dedicated to Delta Lake, outside this guide's scope.

Assuming the 2026 convergence means "it no longer matters which one you pick, they're the same." What happens: someone concludes, from the convergence section, that Iceberg and Delta Lake are interchangeable today, with no practical difference. Why it happens: the convergence evidence is real and compelling, and it's easy to over-generalize it. How to spot it: if your conclusion is "you don't need to know which is which," reread the metadata-mechanism section — today, in 2026, they're still two genuinely distinct on-disk structures (tree vs. flat log), and the proposed convergence for Delta 5.0 is still, as of this lesson's writing, a proposal, not a done deal. How to fix it: the correct reading of the convergence isn't "it no longer matters" — it's "the decision of which to use depends less and less on what each one is technically capable of, and more and more on which catalog and which engine your organization already has," exactly the phrase that closes this lesson's convergence section.

Exercises

Exercise 1 — Fill in the equivalence table yourself, from memory, without looking back. For each row — metadata structure, time travel syntax, small files, old snapshots/versions — write the exact name of the concept or command in Iceberg and in Delta Lake.

See solution

Metadata structure: tree (metadata.json → manifest list → manifest files) in Iceberg, flat log (_delta_log/*.json + Parquet checkpoints) in Delta Lake. Time travel: VERSION AS OF <snapshot-id> / TIMESTAMP AS OF in both, with snapshot-id (a long integer, system-assigned) in Iceberg and a sequential version number (0, 1, 2, ...) in Delta Lake. Small files: rewrite_data_files in Iceberg, OPTIMIZE <table> in Delta Lake. Old snapshots/versions: expire_snapshots + remove_orphan_files in Iceberg (two separate operations), VACUUM <table> [RETAIN num HOURS] in Delta Lake (a single operation covering both cases).

Exercise 2 — Explain, in your own words, why Delta Lake needs Parquet checkpoints and Iceberg's tree doesn't need an equivalent mechanism. Think about what each system would have to do to answer "what are this table's current data files, right now?"

See solution

In Iceberg, the question "what are the current files?" gets answered by reading a single metadata.json (which points at the current snapshot's manifest list, which points at the manifest files, which enumerate the files) — no matter how many snapshots exist in the table's complete history, the answer is always the same reading distance away, because every snapshot is self-contained. In Delta Lake, without a checkpoint, the only way to reconstruct the current state would be to read, in order, every JSON file in the _delta_log since commit zero, accumulating every add and every remove — work that grows without limit as the table accumulates commits. The Parquet checkpoint, generated every ten commits, is how Delta Lake avoids that growing cost: instead of reading thousands of JSON files, a reader only needs the most recent checkpoint plus, at most, the few JSON files after it — a mechanism Iceberg doesn't need to replicate, because its tree structure already has that "constant reading distance" property built into the design.

Exercise 3 — Prediction: if Delta 5.0 does adopt Iceberg v4's metadata tree, as Databricks proposed, what would happen to this lesson's comparison table's "Metadata structure" row? Think about what it would concretely mean for both formats to share the same on-disk structure.

See solution

That row would stop having two distinct answers — it would come to describe a single on-disk structure (Iceberg v4's metadata tree), readable and writable by clients of both formats with no translation layer in between, exactly as this lesson's convergence section literally quotes ("one on-disk structure readable and writable by both formats' clients with no translation layer"). In that scenario, the real difference between "a Delta table" and "an Iceberg table" would stop being about how the metadata is organized on disk, and would become, almost entirely, about which catalog registers it and which provider-specific tool ecosystem surrounds it — the same point the convergence section already anticipates about why the choice increasingly depends on the catalog and the engine, not on the format's technical capability.

Summary and next step

In this lesson you named, for the first and only time in this whole guide, Delta Lake — installing nothing, running not a single line of code against it. You contrasted its metadata mechanism (flat log of JSON commits + Parquet checkpoints) against Iceberg's tree, which this guide's module 2 already taught in depth, confirmed its time travel syntax (VERSION AS OF/TIMESTAMP AS OF) is, in spirit, nearly identical to Iceberg's, and mapped its maintenance operations (OPTIMIZE, VACUUM) against the three this module worked on Iceberg. You closed with the 2026 convergence evidence — Delta UniForm, the Delta 5.0 proposal, Unity Catalog speaking REST — confirming why "which format?" stopped being, in 2026, a functionality question.

Before moving on you should be able to: name the central difference between Iceberg's metadata mechanism and Delta Lake's; and explain, with the evidence quoted in this lesson, why the market names both formats almost interchangeably in 2026.

Lesson 8 closes this module with a project that integrates lessons 3 through 6's real operations — rebuilding the accumulated history, real expire_snapshots, orphan file verification — into a single script with automated asserts, plus a final summary of what really ran and what stayed documented as representative across this whole module.

Resources

• Delta Lake — official documentation, "Table batch reads and writes," source of VERSION AS OF/TIMESTAMP AS OF's and option("versionAsOf", ...)/option("timestampAsOf", ...)'s exact syntax. docs.delta.io/latest/delta-batch.html. In English.
• Delta Lake — official documentation, "Table utility commands," source of OPTIMIZE's and VACUUM ... RETAIN num HOURS's syntax, including the default retention value. docs.delta.io/delta-utility. In English.
• Alex Merced (DEV Community) — "Lakehouse Table Formats in 2026: Iceberg, Delta Lake, Hudi, Paimon, and DuckLake," source of the literal quotes about Delta UniForm, the Delta 5.0 proposal, and this lesson's convergence verdict. dev.to/alexmercedcoder/lakehouse-table-formats-in-2026. In English.
• This same guide, module 2, complete lesson — source of Iceberg's metadata → manifest list → manifest files → data files chain this lesson contrasts against Delta's _delta_log. workbook/module-02-anatomy-of-an-iceberg-table/. In Spanish.
• src/paths/data-engineering-ecosystem/VALIDACION.md — the market audit that literally quotes the job posting asking for "Apache Iceberg or Delta Lake" with no distinction between the two. Internal repo document. In Spanish.
• This guide's DESIGN doc — module 7's section, "Delta Lake named once, by contrast." src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.