Adding a column without rewriting data

Description

This lesson creates this module's new table, kiosko.dim_store, with Kiosko's three stores and their three original columns — store_id, store_name, city. It then adds a fourth column, country, with table.update_schema(). You're going to confirm, with real evidence — not a documentation claim — that this operation doesn't touch a single byte of the already-existing Parquet files, and that the three already-loaded rows are left, for now, with country=None. Populating that column with Kiosko's three real countries is lesson 5's job — this lesson deliberately stops before that step, to isolate schema evolution's pure mechanism.

Connection to the module. Lesson 2 demonstrated that an Iceberg write — a data change — is atomic. This lesson introduces the other kind of change Iceberg supports: a schema change, which doesn't even need to touch data files to complete. It's the first time in this guide table.update_schema() shows up — module 3 had already named it, in its Going deeper section about "what counts as a write," but had never executed it.

An analogy: adding the question to the form, without knocking on any door again

Picking up lesson 1's analogy: kiosko.dim_store is, at this point, an already-complete census of three houses — S01, S02, S03 — with a three-question form. This lesson adds a fourth question to the form — country — but sends no one to knock again on any of the three doors. The three already-filed cards are left, for now, with the new question's box blank — not because the census is broken, but because nobody's gone to fill it in yet. The form changed; the already-filed cards didn't.

Worked example: dim_store, and its first schema evolution

Step 1 — Create kiosko.dim_store, with three columns

With the same kiosko catalog from the three previous guides — a SqlCatalog backed by SQLite, warehouse on the local filesystem — declare the original schema:

# create_dim_store.py
import os

from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import NestedField, StringType

warehouse_path = os.path.abspath("kiosko_warehouse")
catalog_db_path = os.path.abspath("kiosko_catalog.db")
os.makedirs(warehouse_path, exist_ok=True)

catalog = load_catalog(
    "kiosko", type="sql",
    uri=f"sqlite:///{catalog_db_path}", warehouse=f"file://{warehouse_path}",
)
catalog.create_namespace("kiosko")

dim_store_schema = Schema(
    NestedField(field_id=1, name="store_id", field_type=StringType(), required=True),
    NestedField(field_id=2, name="store_name", field_type=StringType(), required=True),
    NestedField(field_id=3, name="city", field_type=StringType(), required=True),
)

table = catalog.create_table("kiosko.dim_store", schema=dim_store_schema)
print("Table created:", table.name())
print()
print(table.schema())

What to expect (verified by running the actual script):

Table created: ('kiosko', 'dim_store')

table {
  1: store_id: required string
  2: store_name: required string
  3: city: required string
}

Step 2 — Load the three stores, and capture snap_before_evolution

# load_stores.py -- Kiosko's three stores, no evolution yet
import os

import pyarrow as pa
from pyiceberg.catalog import load_catalog

warehouse_path = os.path.abspath("kiosko_warehouse")
catalog_db_path = os.path.abspath("kiosko_catalog.db")

catalog = load_catalog(
    "kiosko", type="sql",
    uri=f"sqlite:///{catalog_db_path}", warehouse=f"file://{warehouse_path}",
)
table = catalog.load_table("kiosko.dim_store")

dim_store_pa_schema = pa.schema([
    pa.field("store_id", pa.string(), nullable=False),
    pa.field("store_name", pa.string(), nullable=False),
    pa.field("city", pa.string(), nullable=False),
])

DIM_STORE = [
    {"store_id": "S01", "store_name": "Kiosko Centro", "city": "Bogota"},
    {"store_id": "S02", "store_name": "Kiosko Norte", "city": "Lima"},
    {"store_id": "S03", "store_name": "Kiosko Sur", "city": "Santiago"},
]

pa_table = pa.Table.from_pylist(DIM_STORE, schema=dim_store_pa_schema)
table.append(pa_table)

# captured IMMEDIATELY, following module 3's rule -- never hardcode a snapshot-id
snap_before_evolution = table.current_snapshot().snapshot_id

print("table.scan().to_arrow() after the load:")
for row in table.scan().to_arrow().to_pylist():
    print(f"  {row['store_id']}  {row['store_name']:<15} {row['city']}")
print()
print("snap_before_evolution captured, type:", type(snap_before_evolution).__name__)
print("table.history() has", len(table.history()), "entry(entries)")

What to expect (verified by running the actual script; snap_before_evolution is an integer Iceberg assigns at commit time, different on every run — never hardcoded):

table.scan().to_arrow() after the load:
  S01  Kiosko Centro   Bogota
  S02  Kiosko Norte    Lima
  S03  Kiosko Sur      Santiago

snap_before_evolution captured, type: int
table.history() has 1 entry(entries)

Keep snap_before_evolution: it's dim_store's photo before any schema evolution — the one this module's lesson 6 is going to read, to confirm it's still seen with its original three-column schema, even after the table changes.

Step 3 — table.update_schema().add_column("country", ...), and the proof that no file gets touched

# add_country_column.py -- the schema evolution itself
import os

from pyiceberg.catalog import load_catalog
from pyiceberg.types import StringType

warehouse_path = os.path.abspath("kiosko_warehouse")
catalog_db_path = os.path.abspath("kiosko_catalog.db")

catalog = load_catalog(
    "kiosko", type="sql",
    uri=f"sqlite:///{catalog_db_path}", warehouse=f"file://{warehouse_path}",
)
table = catalog.load_table("kiosko.dim_store")

files_before = sorted(f["file_path"].split("/")[-1] for f in table.inspect.files().to_pylist())
snapshots_before = len(table.history())

with table.update_schema() as update:
    update.add_column("country", StringType())

print("Schema after add_column('country', StringType()):")
print(table.schema())
print()

files_after = sorted(f["file_path"].split("/")[-1] for f in table.inspect.files().to_pylist())
snapshots_after = len(table.history())

print("Data files BEFORE the add_column:", files_before)
print("Data files AFTER the add_column:", files_after)
print("Same files (no Parquet rewritten):", files_before == files_after)
print()
print("table.history() BEFORE:", snapshots_before, " AFTER:", snapshots_after,
      "-- add_column does NOT create a data snapshot")
print()
print("table.scan().to_arrow() -- existing rows after adding the column:")
for row in table.scan().to_arrow().to_pylist():
    print(f"  {row['store_id']}  {row['store_name']:<15} {row['city']:<10} country={row['country']}")

What to expect (verified by running the actual script):

Schema after add_column('country', StringType()):
table {
  1: store_id: required string
  2: store_name: required string
  3: city: required string
  4: country: optional string
}

Data files BEFORE the add_column: ['00000-0-977f7238-b846-4c56-955e-12ac07baf5b5.parquet']
Data files AFTER the add_column: ['00000-0-977f7238-b846-4c56-955e-12ac07baf5b5.parquet']
Same files (no Parquet rewritten): True

table.history() BEFORE: 1  AFTER: 1 -- add_column does NOT create a data snapshot

table.scan().to_arrow() -- existing rows after adding the column:
  S01  Kiosko Centro   Bogota     country=None
  S02  Kiosko Norte    Lima       country=None
  S03  Kiosko Sur      Santiago   country=None

(The Parquet file's exact name is your own run's, different each time — what matters is that the before and after lists are identical.)

Three facts, all confirmed with evidence, not a promise: 1) the schema now has four columns, with country marked optional (unlike the other three, required) — you're going to see why this is mandatory in this lesson's Going deeper section. 2) the data-file list is exactly the same before and after — the same three-row Parquet file, not one byte rewritten. 3) table.history() gained no new entry — add_column isn't a data write, so it produces no snapshot, consistent with the distinction module 3 already previewed. And the three existing rows, read after the change, show country=None — this lesson's analogy's "blank box."

Diagram: what changes, and what doesn't

flowchart LR
    A["kiosko.dim_store\n3 columns, 3 rows\n1 Parquet file"] -->|"table.update_schema()\nadd_column('country')"| B["kiosko.dim_store\n4 columns, 3 rows\nSAME Parquet file"]
    A -.->|"snapshots"| S1["1 snapshot\n(the append)"]
    B -.->|"snapshots"| S1
    B -->|"country for the 3 rows"| N["None, None, None\n(pending population -- lesson 5)"]

The only file that changes is the metadata one — a new JSON, with the four-column schema and a reference to the same snapshot and the same manifest as always. The data file — the Parquet with the three rows — never gets touched.

Going deeper: why the new column has to be optional

Notice a detail in the resulting schema: country was marked optional, while store_id, store_name, and city are required. This isn't a style choice — it's a mandatory consequence of what this lesson just demonstrated. If country were required with no default value, the three already-existing rows would immediately need a real value there — but this lesson proved add_column touches no data file, so there's no mechanism to fill that value into the old rows at the exact instant the column gets added. Apache Iceberg's official documentation, on its "Evolution" page, sums it up like this: "Iceberg schema updates are metadata changes, so no data files need to be rewritten to perform the update." A required column with no data rewritten is, literally, a contradiction — that's why PyIceberg requires, by default, that any column added to a table with existing rows be optional (required=False is add_column()'s default), unless you give it an explicit default_value Iceberg can use to fill in the old rows without writing anything.

The same official documentation is explicit about the underlying guarantee, under the heading "Correctness": "Added columns never read existing values from another column [...] Iceberg uses unique IDs to track each column in a table. When you add a column, it is assigned a new ID so existing data is never used by mistake." That's the technical reason country got field_id=4 in this lesson's schema, never reusing the 1, 2, or 3 the original columns already had — the same field_id mechanism this guide's module 1 already introduced as the backbone of how Iceberg identifies each piece of data, now applied to why a new column can never, by accident, inherit values from an old one.

Common mistakes

Calling update.add_column() outside the with table.update_schema() as update: block. What happens: someone writes update = table.update_schema(), then update.add_column(...), but never commits, and the table's schema never changes — with no visible error warning about the problem. Why it happens: update_schema() returns an UpdateSchema object that accumulates changes, but doesn't apply them until something confirms the transaction; without the with, that "something" never happens. How to spot it: if after calling add_column() you print table.schema() again and it still shows the old columns, your change was never committed. How to fix it: always use the with table.update_schema() as update: update.add_column(...) pattern — the with block is what calls commit() automatically on exit, the exact same context-manager pattern you already used, without knowing it, in any Python with open(...) as f:.

Expecting country to have the real country values immediately after add_column. What happens: someone runs step 3 of this lesson, sees country=None on all three rows, and concludes something went wrong. Why it happens: it's natural to expect "adding a column derived from another" to complete the calculation in the same step. How to spot it: if your goal is to see country='Colombia' for S01 right after this step, revisit lesson 1's map — really populating country is, on purpose, a separate step. How to fix it: nothing to fix in this lesson — country=None for all three existing rows is step 3's correct, expected result. This module's lesson 5 does the real population, with a normal overwrite(), exactly the way module 3 populated the P002 change.

Exercises

Exercise 1 — Reproduce the three steps yourself, and confirm the identical file list. In a new directory, run this lesson's three scripts in order. Confirm files_before and files_after from step 3 are exactly the same list (a single Parquet file), and that table.history() stays at 1 after the add_column.

See solution

If you followed the three steps in the same directory, your output should structurally match this lesson's: the Parquet file's exact name is going to be different (it includes a UUID generated on your own run), but files_before == files_after should give True, and table.history() should report 1 both before and after the add_column. If you see 2 after the add_column, check whether you accidentally called table.append() or table.overwrite() at some intermediate point.

Exercise 2 — Prediction: what would happen if you tried to add country as a required column, with no default_value? Without running it yet, look up PyIceberg's API reference (add_column(path, field_type, doc=None, required=False, default_value=None)) for what would happen if you called update.add_column("country", StringType(), required=True) on a table that already has three loaded rows, with no default_value specified.

See solution

PyIceberg rejects that operation with an explicit error, because adding a required column with no default value would violate this lesson's Going deeper guarantee: the three existing rows would have no valid value for a column the schema says must always have a value. The only way to add a required column to a table with data already loaded is to provide a default_value Iceberg can apply, at the metadata level, to the existing rows — with no need to physically rewrite them, but in a way that any reader knows what value to assume for them.

Exercise 3 — Explain, in your own words, why country's field_id is 4 and not 1, 2, or 3. In 2-3 sentences, using this lesson's official documentation quote, explain why Iceberg never reuses an already-used field_id, even if that number corresponded to a column that no longer exists.

See solution

Iceberg identifies each column by its field_id, not by its name or its position — it's the mechanism, already introduced in this guide's module 1, that makes it possible to rename or reorder columns unambiguously. If a field_id were ever reused, an old Parquet file using that number to refer to a different column could, by accident, "resurrect" with data that never had any relationship to the new column — exactly the kind of silent error the official quote describes: "so existing data is never used by mistake." Always assigning the next available number, never recycling any, is the simplest way to guarantee that never happens.

Summary and next step

In this lesson you created kiosko.dim_store with its three original columns, captured snap_before_evolution before any change, and added the country column with table.update_schema(). You confirmed, with real evidence — the same Parquet-file list before and after, the same snapshot count — that this operation is purely metadata: it doesn't rewrite, doesn't touch, doesn't create any new data file. And you saw why the new column has to be optional, and why it gets a field_id that was never reused.

Before moving on you should be able to: add a column to an existing Iceberg table with table.update_schema(); explain, with your own evidence, why that operation doesn't rewrite data files; and explain why a column added to a table with existing rows must be optional unless you declare a default_value.

kiosko.dim_store now has four columns, with country still empty. Lesson 4 adds two more operations to the same mechanism — rename_column() and delete_column() — with a practice column, temp_notes, added and dropped within the same lesson.

Resources

• Apache Iceberg — official documentation, "Evolution," "Schema evolution" and "Correctness" sections, source of this lesson's two quotes about metadata changes and unique field IDs. iceberg.apache.org/docs/latest/evolution. In English.
• PyIceberg — API reference, table.update_schema() and UpdateSchema.add_column(path, field_type, doc=None, required=False, default_value=None). py.iceberg.apache.org/api. In English.
• data-engineering-foundations-guide DESIGN doc — source of stores's exact schema (store_id, store_name, city) this lesson loads unchanged, before adding country. src/guides/data-engineering-foundations-guide/DISENO.md. In Spanish.
• This guide's DESIGN doc — the "Schema evolution" section (M4), the exact source of the dim_store → snap_before_evolution → add_column('country') sequence this lesson runs. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.