Renaming and dropping columns safely

Description

Lesson 3 demonstrated that adding a column is a pure metadata operation. This lesson extends that same guarantee to two other operations: renaming a column (rename_column) and dropping it (delete_column). You're going to rename store_name back and forth, confirming no Parquet file gets touched even when a column's visible name changes twice. And you're going to add a practice column, temp_notes, only to drop it within the same lesson — the direct proof that a DROP COLUMN in Iceberg doesn't rewrite any data either, something that surprises anyone coming from a system where dropping a column is an expensive operation.

Connection to the module. Lesson 3 left kiosko.dim_store with four columns, country still empty. This lesson doesn't touch country at all — that's lesson 5's job — instead, it uses the rest of the table as the setting to complete update_schema()'s mechanism: you already know add_column, this lesson adds rename_column and delete_column to the same vocabulary.

An analogy: changing a drawer's label, and emptying one that's no longer needed

Think of a physical filing cabinet, with labeled drawers. Renaming a column is like peeling off a drawer's old label and sticking on a new one — "Store Name" instead of "store_name," say — without touching a single folder inside: the drawer's content is exactly the same, only the paper stuck outside changed. Dropping a column is different, but just as simple from the whole cabinet's point of view: a drawer no longer needed gets removed from the index — nobody looks there again — but that doesn't force emptying, burning, or reorganizing the rest of the cabinet. The other drawers, with their own labels, stay exactly where they were.

Worked example: renaming back and forth, and adding-and-dropping in the same lesson

Step 1 — rename_column: store_name → outlet_name, and back

# rename_and_scratch_column.py
import os

from pyiceberg.catalog import load_catalog
from pyiceberg.types import StringType

warehouse_path = os.path.abspath("kiosko_warehouse")
catalog_db_path = os.path.abspath("kiosko_catalog.db")

catalog = load_catalog(
    "kiosko", type="sql",
    uri=f"sqlite:///{catalog_db_path}", warehouse=f"file://{warehouse_path}",
)
table = catalog.load_table("kiosko.dim_store")

files_before = sorted(f["file_path"].split("/")[-1] for f in table.inspect.files().to_pylist())
snapshots_before = len(table.history())

with table.update_schema() as update:
    update.rename_column("store_name", "outlet_name")

print("Schema after rename_column('store_name', 'outlet_name'):")
print(table.schema())
print()
print("scan() after the rename -- the DATA didn't change, only the column name:")
for row in table.scan().to_arrow().to_pylist():
    print(" ", row)

files_after_rename = sorted(f["file_path"].split("/")[-1] for f in table.inspect.files().to_pylist())
print()
print("Identical files after the rename:", files_before == files_after_rename)
print("Snapshots before:", snapshots_before, "-- after:", len(table.history()))

with table.update_schema() as update:
    update.rename_column("outlet_name", "store_name")
print()
print("Schema after reverting the rename (back to store_name):")
print(table.schema())

What to expect (verified by running the actual script):

Schema after rename_column('store_name', 'outlet_name'):
table {
  1: store_id: required string
  2: outlet_name: required string
  3: city: required string
  4: country: optional string
}

scan() after the rename -- the DATA didn't change, only the column name:
  {'store_id': 'S01', 'outlet_name': 'Kiosko Centro', 'city': 'Bogota', 'country': None}
  {'store_id': 'S02', 'outlet_name': 'Kiosko Norte', 'city': 'Lima', 'country': None}
  {'store_id': 'S03', 'outlet_name': 'Kiosko Sur', 'city': 'Santiago', 'country': None}

Identical files after the rename: True
Snapshots before: 1 -- after: 1

Schema after reverting the rename (back to store_name):
table {
  1: store_id: required string
  2: store_name: required string
  3: city: required string
  4: country: optional string
}

Notice the second column's field_id: it's 2 before the rename, 2 during (as outlet_name), and 2 again once back to store_name. The visible name changed twice; the internal identifier — the one that actually connects the schema to the Parquet's physical data — never moved. That's what makes renaming safe: any data file written before the rename still gets read correctly afterward, because it never depended on the text name, only on the field_id.

Step 2 — temp_notes: added, and dropped, within the same lesson

with table.update_schema() as update:
    update.add_column("temp_notes", StringType())
print("\nSchema after add_column('temp_notes'):")
print(table.schema())

files_after_temp_notes = sorted(f["file_path"].split("/")[-1] for f in table.inspect.files().to_pylist())
print("Identical files after adding temp_notes:", files_before == files_after_temp_notes)

with table.update_schema() as update:
    update.delete_column("temp_notes")
print("\nSchema after delete_column('temp_notes') -- the DROP:")
print(table.schema())

files_after_drop = sorted(f["file_path"].split("/")[-1] for f in table.inspect.files().to_pylist())
print("Identical files after the DROP:", files_before == files_after_drop)
print("Total snapshots after rename + add + drop (all schema operations, zero data ones):",
      len(table.history()))

What to expect (verified by running the actual script, continuing on the same table from step 1):

Schema after add_column('temp_notes'):
table {
  1: store_id: required string
  2: store_name: required string
  3: city: required string
  4: country: optional string
  5: temp_notes: optional string
}
Identical files after adding temp_notes: True

Schema after delete_column('temp_notes') -- the DROP:
table {
  1: store_id: required string
  2: store_name: required string
  3: city: required string
  4: country: optional string
}
Identical files after the DROP: True
Total snapshots after rename + add + drop (all schema operations, zero data ones): 1

temp_notes came to exist in the schema, with field_id=5, and disappeared completely without the rename, the add, or the drop moving the snapshot count or touching this table's single Parquet file — it's still 1 since lesson 3. Four different schema operations — rename, rename back, add, drop — all committed, and the data-write history didn't move once.

Step 3 — field_id=5 never gets reused

from pyiceberg.types import StringType

with table.update_schema() as update:
    update.add_column("scratch_check", StringType())

print("After adding a new column AFTER dropping temp_notes (which used field_id=5):")
for f in table.schema().fields:
    print(f"  field_id={f.field_id}  name={f.name}")

with table.update_schema() as update:
    update.delete_column("scratch_check")

What to expect (verified by running the actual script):

After adding a new column AFTER dropping temp_notes (which used field_id=5):
  field_id=1  name=store_id
  field_id=2  name=store_name
  field_id=3  name=city
  field_id=4  name=country
  field_id=6  name=scratch_check

scratch_check got field_id=6, skipping the 5 temp_notes already used and left behind. Iceberg never reuses a field_id, not even after the original column gets completely dropped — exactly the same guarantee lesson 3 already demonstrated for columns added over existing data, applied here to columns that no longer exist.

Diagram: four schema operations, zero data snapshots

flowchart LR
    A["dim_store\n4 columns (with country=None)\nfield_id 1..4\n1 snapshot"] -->|"rename_column\nstore_name -> outlet_name"| B["same field_id=2,\nnew name"]
    B -->|"rename_column\noutlet_name -> store_name"| C["field_id=2,\noriginal name back"]
    C -->|"add_column\ntemp_notes"| D["new field_id=5,\n5 columns"]
    D -->|"delete_column\ntemp_notes"| E["field_id=5 retired,\n4 columns again"]
    E -.->|"1 snapshot total,\nno change"| F["table.history()\nstill at 1"]

Going deeper: why dropping a column doesn't free space right away

It's worth being precise about what it physically means for delete_column("temp_notes") to "not rewrite data." If temp_notes had ended up with real values written to the Parquet — this lesson dropped it before populating it, on purpose, to isolate the mechanism — those values would still physically exist inside the Parquet file after the DROP. What changes is that the current schema no longer projects that column: any new read (table.scan()) simply ignores that portion of the file, as if it didn't exist. The disk space those old values occupy isn't reclaimed by delete_column — it's reclaimed, if needed, with an explicit maintenance operation like compaction (rewrite_data_files), which module 7 of this guide teaches. This distinction — "the schema stops seeing the column" versus "the disk stops holding the bytes" — is the same one you already saw in module 3, when an overwrite() "deleted" rows without immediately deleting the Parquet file that held them: in both cases, Iceberg prioritizes not touching existing files over immediately freeing space, and leaves physical cleanup for a separate, deliberate maintenance step.

Common mistakes

Assuming delete_column in Iceberg is as expensive as in a system that does rewrite files. What happens: someone, familiar with an engine where dropping a column from a large table triggers a full rewrite that can take hours, avoids adding practice columns in Iceberg "just in case," thinking cleaning up later is going to be expensive. Why it happens: in many traditional relational systems, and in plain Parquet with no table format on top, changing an already-written file's schema does require rewriting it entirely. How to spot it: if you avoid experimenting with add_column/delete_column on an Iceberg table out of fear of the cost, revisit this lesson's evidence — the snapshot count and the file list didn't change once across the four operations. How to fix it: in Iceberg, adding and dropping columns are metadata operations, practically instant regardless of how many rows the table has — the cost of rewriting data, if it's ever needed, is a separate, explicit decision (compaction, module 7), not an automatic consequence of evolving the schema.

Confusing "the field_id is never reused" with "column names can never repeat." What happens: someone, after dropping temp_notes, tries to add back a column with the same name temp_notes and expects it to fail, thinking the name is also "burned" like the field_id. Why it happens: it's easy to generalize the field_id's "never reused" rule to the visible name, which is what most people notice first. How to spot it: if you avoid reusing an already-dropped column name for fear of a conflict, try adding it again. How to fix it: a column's name really can be freely reused after a DROP — Iceberg simply assigns a new, higher field_id to that "new" column with the repeated name; the only thing that's never reused is the internal number. scratch_check, in step 3 of this lesson, could have been named temp_notes again with no problem at all, and would have gotten field_id=6 all the same.

Exercises

Exercise 1 — Reproduce the four operations yourself, and confirm each column's field_id at the end. With kiosko.dim_store in the state lesson 3 left, run this lesson's three steps. Confirm the final schema has exactly four columns (store_id, store_name, city, country), with field_ids 1, 2, 3, 4 — with no trace at all of temp_notes or scratch_check.

See solution

If you started from lesson 3's exact state, your final schema should match this lesson's: four columns, field_ids 1 through 4 in order, store_name back to its original name. The next column's field_id you add (if you try) is going to be 7, not 5 or 6 — because both temp_notes (5) and scratch_check (6) already "burned" those numbers in this lesson.

Exercise 2 — Prediction: what would happen if you tried to rename store_id to city, when city already exists? Without running it yet, predict: if you called update.rename_column("store_id", "city") on dim_store's current schema — which already has a column named city — what do you expect to happen?

See solution

PyIceberg rejects that operation, because it would produce two columns with the same visible name within the same schema — an ambiguity no query engine could reliably resolve (which city does a query mentioning it refer to?). The "every column has a unique field_id" guarantee doesn't eliminate the need for names, within the same schema level, to stay unique; rename_column checks this before accepting the change, exactly the way adding a column with an already-used name would.

Exercise 3 — Explain, in your own words, why this lesson reverts store_name's rename before moving on. In 1-2 sentences, explain why step 1 of this lesson leaves dim_store with the store_name name restored, instead of continuing the rest of the module with outlet_name.

See solution

The rename's purpose in this lesson is purely demonstrative — proving the mechanism is safe and doesn't touch data — not a real business change the rest of this guide needs. Reverting it keeps kiosko.dim_store with the column name lessons 5 through 8 — and the rest of Kiosko's ecosystem, which always used store_name — expect to find, preventing an experiment from this lesson from introducing a naming inconsistency into the rest of the module.

Summary and next step

In this lesson you extended table.update_schema()'s vocabulary with rename_column and delete_column, on top of the previous lesson's add_column. You confirmed, with the same discipline of comparing file lists and snapshot counts, that renaming a column back and forth, and adding-and-dropping a practice column, touch no data file and create no new snapshot. And you saw, with direct evidence, that a dropped column's field_id — temp_notes, with field_id=5 — never gets reused, not even for a column with the same name.

Before moving on you should be able to: rename and drop a column on an Iceberg table with table.update_schema(); explain why a dropped field_id is never reused; and tell apart "the schema stops projecting a column" from "the disk frees the space that column occupied."

kiosko.dim_store is back to its four clean columns, with country still None for all three stores. Lesson 5 does the real work: deterministically populating country from city, with Kiosko's three exact countries.

Resources

• Apache Iceberg — official documentation, "Evolution," "Schema evolution" and "Correctness" sections, the formal source for the rename and delete guarantees this lesson runs. iceberg.apache.org/docs/latest/evolution. In English.
• PyIceberg — API reference, UpdateSchema.rename_column(path_from, new_name) and UpdateSchema.delete_column(path). py.iceberg.apache.org/api. In English.
• This guide's DESIGN doc — the "Schema evolution" section (M4), the exact source of temp_notes as a column added and dropped within the same lesson, to demonstrate a DROP COLUMN doesn't rewrite data. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.