What "ACID" actually guarantees in this exact context

Description

This lesson takes a step back. Lessons 2 through 6 of this module demonstrated, over and over, that Iceberg writes are atomic and that its reads never see a halfway state. It's tempting, with that accumulated evidence, to say "Iceberg is ACID" and leave it there — but that phrase, without precision, can suggest a guarantee Iceberg doesn't give: that of a full relational transactional engine, with multi-table transactions and explicit ROLLBACK. This lesson builds, with real code, the exact scenario where Iceberg's guarantee comes into play — two writers competing for the same commit — and draws, with the same precision, the line for exactly how far that guarantee reaches and where it ends.

Connection to the module. Lesson 2 demonstrated half of this story: that no external reader sees an intermediate state during a write. This lesson demonstrates the other half: what happens when two writers — not a writer and a reader — try to modify the same table at the same time. It's the same "stage, then commit" mechanism lesson 2 already introduced, now seen from the angle of concurrency between writes, not between a write and a read.

An analogy: the front desk, with two people asking for the same slot

Go back to lesson 2's government-office front desk. Now imagine two people arrive, almost at the same time, to update the same record — each prepared, on their own, their own new document, with no idea the other was doing the same thing. The desk can't accept both changes at once, because each started from a different version of the original record, and applying both with no coordination would produce a result nobody asked for. The desk's solution is simple and strict: the first person who arrives with their complete document wins the slot, their change gets applied; the second, when they present theirs, discovers the record already changed since they started preparing it, and the desk tells them, unambiguously, "this is no longer valid — go prepare it again with the updated record." Nobody silently loses their work. Nobody ends up with a mixed-together record, half of one change and half of the other. The desk clearly rejects whoever arrived late to the race.

Worked example: two writers, one wins, the other is rejected with evidence

Step 1 — Two processes load the same table, at the same time

This illustration uses its own, disposable catalog and table, isolated from kiosko.dim_store — the goal is to show concurrency's pure mechanism, not mix the experiment with Kiosko's real state.

# concurrent_write_conflict.py -- isolated illustration, NOT part of Kiosko's model
import os
import shutil

from pyiceberg.catalog import load_catalog
from pyiceberg.exceptions import CommitFailedException
from pyiceberg.schema import Schema
from pyiceberg.types import NestedField, StringType

demo_warehouse = os.path.abspath("acid_demo_warehouse")
demo_db = os.path.abspath("acid_demo_catalog.db")
os.makedirs(demo_warehouse, exist_ok=True)

demo_catalog = load_catalog(
    "acid_demo", type="sql",
    uri=f"sqlite:///{demo_db}", warehouse=f"file://{demo_warehouse}",
)
demo_catalog.create_namespace("acid_demo")

schema = Schema(NestedField(field_id=1, name="ticket_id", field_type=StringType(), required=True))
demo_catalog.create_table("acid_demo.race_table", schema=schema)

# two different "processes" load the SAME table handle at the initial instant --
# each one has its own in-memory copy of the metadata that's current right now
table_process_a = demo_catalog.load_table("acid_demo.race_table")
table_process_b = demo_catalog.load_table("acid_demo.race_table")

print("schema_id both processes see when loading the table:",
      table_process_a.schema().schema_id, table_process_b.schema().schema_id)

What to expect (verified by running the actual script):

schema_id both processes see when loading the table: 0 0

Both processes start from exactly the same point — the same schema_id=0, the freshly created table, with no difference at all between what each one sees. Neither knows, yet, that the other exists.

Step 2 — Process A commits first; process B arrives late

with table_process_a.update_schema() as update:
    update.add_column("status", StringType())
print("\nProcess A commits first -- adds the 'status' column. OK.")
print("Current schema_id after A's commit:", table_process_a.schema().schema_id)

try:
    with table_process_b.update_schema() as update:
        update.add_column("priority", StringType())
    print("Process B committed with no error (this should NOT happen)")
except CommitFailedException as e:
    print(f"\nProcess B fails with CommitFailedException:\n  {e}")

What to expect (verified by running the actual script; the exception's exact text is from PyIceberg 0.11.1, and it's fully reproducible: the schema_id numbers in this isolated experiment are deterministic, unlike a snapshot_id):

Process A commits first -- adds the 'status' column. OK.
Current schema_id after A's commit: 1

Process B fails with CommitFailedException:
  Requirement failed: current schema id has changed: expected 0, found 1

There's the guarantee, with literal evidence, not a documentation promise. Process A, which started from schema_id=0, committed first, and the table advanced to schema_id=1. Process B, which also started from schema_id=0 but came in second to commit, got rejected — not with corrupted data, not with a halfway status-and-priority mixed result, but with a clear exception that states, precisely, what it expected and what it found. The record never ended up with a Frankenstein schema of "some of A, some of B" — either A won completely, or (had B arrived first) B would have won completely. Never both, partially.

Diagram: the race, and the referee

sequenceDiagram
    participant A as Process A
    participant Cat as Catalog (kiosko)
    participant B as Process B

    A->>Cat: load_table() -- sees schema_id=0
    B->>Cat: load_table() -- sees schema_id=0
    A->>A: prepares new metadata (schema_id=1)
    A->>Cat: UPDATE ... WHERE expected_schema_id=0
    Cat-->>A: OK -- current is now schema_id=1
    B->>B: prepares new metadata (based on schema_id=0, now stale)
    B->>Cat: UPDATE ... WHERE expected_schema_id=0
    Cat-->>B: REJECTED -- current is already schema_id=1, not 0
    B->>B: CommitFailedException

Going deeper: the three letters that do apply, and the one that doesn't apply without nuance

It's worth being precise, letter by letter, about what "ACID" — Atomicity, Consistency, Isolation, Durability — this module demonstrated, and with what exact scope:

• Atomicity — demonstrated in lesson 2: an Iceberg write applies completely or doesn't apply at all, never halfway. Confirmed with 490 concurrent reads capturing no intermediate state.
• Isolation — demonstrated in this lesson: two writers competing for the same commit never produce a mixed-together result; one wins, the other gets rejected with a clear exception (CommitFailedException), and can retry over the new state. Apache Iceberg's official documentation calls this, with that exact name, "serializable isolation": "Commits replace the path of the current table metadata file using an atomic operation. This ensures that all updates to table data and metadata are atomic, and is the basis for serializable isolation."
• Durability — inherited from the underlying storage: once the catalog's UPDATE commits, the metadata file and the data files it references are already written, complete, to disk (or to object storage, in a real deployment) — they don't depend on any process still running to persist.
• Consistency — this is where you need to be more precise. Iceberg guarantees that one table never ends up in a state inconsistent with itself — its metadata always correctly describes its own files. What Iceberg does not guarantee is consistency across different tables: if a script writes first to kiosko.dim_store and then fails writing to kiosko.fact_orders, the first write stays committed, complete, with no automatic mechanism undoing it. There's no ROLLBACK spanning both tables at once, because every Iceberg table has its own commit catalog, independent of any other's.

That's why the precise claim isn't "Iceberg is a full ACID transactional engine" — it is, in this lesson's exact words: Iceberg guarantees atomicity and serializable isolation for a single table's changes, with optimistic concurrency to resolve conflicts between writers. That's exactly what you needed to trust that table.overwrite() (module 3) and table.update_schema() (this module) never leave kiosko.dim_store halfway through — and it's also, exactly, where the guarantee ends, without stretching it further than the official documentation backs up.

Common mistakes

Designing a pipeline that depends on an automatic ROLLBACK between two different Iceberg tables. What happens: someone writes a script that updates kiosko.dim_store and then kiosko.fact_orders, and assumes that if the second write fails, the first undoes itself. Why it happens: in a relational database with multi-table transactions, that's exactly the expected behavior, and it's natural to assume the same here. How to spot it: if your script doesn't explicitly handle what to do when the second of two Iceberg writes fails after the first already committed, you have this risk. How to fix it: design each table write as an independent unit, and handle errors explicitly — for example, checking the first write's result before deciding whether the second should run, or building your own compensation mechanism if you genuinely need to revert an already-committed change. This is, precisely, one of the problems a real orchestrator (airflow-and-declarative-orchestration-guide, named without being implemented in this guide) is designed to handle with retries and explicit dependencies between steps.

Thinking CommitFailedException means the data got corrupted. What happens: someone, seeing step 2's exception in this lesson for the first time, panics thinking something went wrong with the table. Why it happens: the exception's name, and the fact that it interrupts the script's execution, can sound like a serious failure. How to spot it: if your reaction to a CommitFailedException is to check the table's integrity, first check what the exception is made of: a clean rejection, with no change applied halfway. How to fix it: CommitFailedException is, exactly, the safety mechanism working as it should — it means Iceberg detected a race and resolved it without corrupting anything, rejecting whoever arrived late. The correct response isn't to worry about the table: it's to reload it (catalog.load_table(...) again, to get the current state) and retry the operation on top of that updated base, if the change is still needed.

Exercises

Exercise 1 — Reproduce the race yourself, and confirm the exception's exact message. Run this lesson's full script. Confirm you see schema_id both processes see when loading the table: 0 0, and that process B fails with a message mentioning "expected 0, found 1".

Exercise 2 — Modify the script so process B successfully retries. After catching the CommitFailedException, add code that reloads the table (demo_catalog.load_table("acid_demo.race_table")) and retries add_column("priority", ...) on that fresh version. Confirm the second attempt succeeds.

try:
    with table_process_b.update_schema() as update:
        update.add_column("priority", StringType())
except CommitFailedException:
    table_process_b_retry = demo_catalog.load_table("acid_demo.race_table")
    with table_process_b_retry.update_schema() as update:
        update.add_column("priority", StringType())
    print("Retry succeeded. Final schema:")
    print(table_process_b_retry.schema())

Exercise 3 — Explain, in your own words, why this lesson says "Iceberg guarantees atomicity and serializable isolation for a single table," instead of simply "Iceberg is ACID." In 2-3 sentences, justify why the full phrase is more precise, using kiosko.dim_store and kiosko.fact_orders as two different tables as your example.

Summary and next step

In this lesson you built, with real code, the exact scenario where Iceberg's isolation guarantee comes into play: two processes competing for the same commit, one wins, the other gets rejected with a clear, reproducible exception — CommitFailedException: Requirement failed: current schema id has changed: expected 0, found 1. You joined that evidence with lesson 2's (atomicity) to precisely state, letter by letter, what "ACID" guarantees in the context of a single Iceberg table, and where that guarantee ends — at a table's boundary, not in multi-table transactions in the style of a full relational engine.

Before moving on you should be able to: explain, in your own words, ACID's four letters in the exact context of an Iceberg table; reproduce a real write conflict and its error message; and explain why Iceberg doesn't offer multi-table transactions, with a concrete kiosko example.

With the complete mechanism demonstrated — atomicity, safe schema evolution, reading the past, and "ACID"'s precise limits — lesson 8 brings this module's seven pieces together into a single project: kiosko.dim_store evolved end to end, with automatic assert statements confirming every claim.

Resources

• Apache Iceberg — official documentation, "Reliability," "Serializable Isolation" and "Concurrent write operations" sections, the formal source for this lesson's two quotes about atomicity and optimistic concurrency. iceberg.apache.org/docs/latest/reliability. In English.
• PyIceberg — API reference, pyiceberg.exceptions.CommitFailedException, the real exception this lesson catches and explains. py.iceberg.apache.org/api. In English.
• This guide's DESIGN doc — the "Schema evolution" section (M4), the exact source of the question this lesson answers: "what does 'ACID' guarantee in this context (write isolation and atomicity, not a full transactional engine)." src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.