Module 7: Catalogs Maintenance And Delta Lake By Contrast

Removing orphan files

Next →

Description

Lesson 5 left an explicit problem, deliberately unsolved: seven physical Parquet files on disk, but only two tracked by any current kiosko.dim_product snapshot. This lesson gives those five files an exact name — orphans — and identifies them with real code, counting exact bytes. And it documents, with the same honesty as lesson 4 (compaction) and the rest of this module, why the operation that would really delete them — remove_orphan_files — doesn't run in pure-Python PyIceberg 0.11.1.

Connection to the module. This lesson is lesson 5's direct continuation, not a new topic. expire_snapshots() rewrote the metadata; this lesson precisely measures what got left behind on the filesystem by that rewrite.

What exactly an orphan file is

A file is orphaned when it meets two conditions at once: it physically exists in the table's warehouse, and no snapshot the current metadata still tracks references it from any manifest file. It isn't the same as "an old file" — a file can be old and still perfectly alive, if some non-expired snapshot still needs it — and it isn't the same as "a corrupted file" — an orphan is usually a perfectly valid Parquet file, just one nobody knows exists anymore. The most common cause, the one that produced this lesson's five files: expiring the snapshots that referenced them, with the expiration operation not also taking care of deleting them from disk — exactly what lesson 5 confirmed about PyIceberg 0.11.1.

An analogy: the toast copies, still in the photo lab's drawer

Going back to the wedding album: expiring lesson 5's redundant snapshots is like crossing out, in the album's index, the nineteen references to the toast's duplicate copies — the index now clearly says only one photo of that moment exists. But the nineteen physical copies are still exactly where the photo lab left them: in a drawer, with no index number pointing at them. Nobody's going to find them by browsing the album — the index no longer mentions them — but they're still there, taking up space, until someone checks the drawer directly and throws them out.

Worked example: counting the orphans with real evidence

Step 1 — Compare what's tracked against what's physical

# find_orphans.py -- identifies kiosko.dim_product's orphan files (READ ONLY)
import os

from pyiceberg.catalog import load_catalog

warehouse_path = os.path.abspath("kiosko_warehouse")
catalog_db_path = os.path.abspath("kiosko_catalog.db")

catalog = load_catalog(
    "kiosko", type="sql",
    uri=f"sqlite:///{catalog_db_path}", warehouse=f"file://{warehouse_path}",
)
table = catalog.load_table("kiosko.dim_product")

tracked = {row["file_path"] for row in table.inspect.all_data_files().select(["file_path"]).to_pylist()}
print("files tracked by some current snapshot:", len(tracked))

data_dir = os.path.join(warehouse_path, "kiosko", "dim_product", "data")
on_disk = {"file://" + os.path.join(data_dir, fn) for fn in os.listdir(data_dir) if fn.endswith(".parquet")}
print("physical .parquet files on disk:", len(on_disk))

orphans = on_disk - tracked
print("\norphan files (on disk, zero current snapshots reference them):", len(orphans))
total_bytes = 0
for f in sorted(orphans):
    local_path = f.replace("file://", "")
    size = os.path.getsize(local_path)
    total_bytes += size
    print(" ", f.split("/")[-1], f"{size} bytes")
print(f"\ntotal orphan bytes: {total_bytes}")

What to expect (verified by running the real script, on the state lesson 5 left behind; the file names are from your own run, different every time — the count and total size are deterministic):

files tracked by some current snapshot: 2
physical .parquet files on disk: 7

orphan files (on disk, zero current snapshots reference them): 5
  <file-1>.parquet 1837 bytes
  <file-2>.parquet 1837 bytes
  <file-3>.parquet 1837 bytes
  <file-4>.parquet 1837 bytes
  <file-5>.parquet 1837 bytes

total orphan bytes: 9185

Five files, 9,185 bytes total — under 9 KB, on this four-row lab table — but the mechanism is identical, whatever the scale: every snapshot you expired in lesson 5 left behind, exactly, the data file it had written at the time, and none of those five files got touched by the expiration operation.

Step 2 — Why this script is diagnostic, never a deletion tool

Notice something deliberate: Step 1 deletes nothing. It calculates, compares, reports — the same read-only discipline this guide applied in module 2 when exploring the warehouse/ on disk. Deleting a file "that looks orphaned" with os.remove(), by hand, outside an official maintenance operation, is dangerous for a concrete reason: this script's comparison was made against the metadata at the exact instant it ran — if another process is, at that same moment, writing a new snapshot that hasn't confirmed its commit yet, its temporary files might transiently appear as "not yet tracked" without being truly orphaned. A truly orphaned file needs to be confirmed against all the current metadata, including that of any write in progress — exactly the work the official remove_orphan_files operation does, and exactly why it exists as an engine operation, with a configurable retention window, instead of as a plain os.remove().

The real operation: remove_orphan_files — not available in pure-Python PyIceberg 0.11.1

It was verified, just like in lesson 4, against PyIceberg 0.11.1's installed source code and against its official API reference: no remove_orphan_files method exists on Table or on table.maintenance. It's, like compaction, a distributed engine operation — in the 2026 Iceberg ecosystem, it runs on Spark, via the remove_orphan_files SQL procedure. The following block is marked as representative:

-- (representative) -- syntax verified against Iceberg's official documentation,
-- NOT executed in this environment: PyIceberg 0.11.1 doesn't implement remove_orphan_files.

-- first, ALWAYS: a dry run -- lists candidates, deletes nothing yet
CALL local.system.remove_orphan_files(table => 'kiosko.dim_product', dry_run => true);

-- after reviewing the list, the version that does delete
CALL local.system.remove_orphan_files(table => 'kiosko.dim_product');

What to expect (representative): dry_run => true would return one row per orphan-candidate file — the same five Step 1 identified, on a real warehouse — without deleting a single byte yet. Run without dry_run, the procedure physically deletes the files confirmed as orphans, and returns a report (orphan_file_location, one row per deleted file) — the same explicit-verification pattern you already saw in rewrite_data_files (lesson 4): never trust an operation did what was expected without reviewing its output report.

The retention window: why "faster" isn't "safer"

Iceberg's official documentation includes an explicit warning about this operation, with a phrase worth quoting in full: "It is dangerous to remove orphan files with a retention interval shorter than the time expected for any write to complete because it might corrupt the table if in-progress files are considered orphaned and are deleted. The default interval is 3 days." This connects directly with what this lesson's Step 2 already warned about with code: a file that looks orphaned at the moment of comparison could actually be a file another write is still building, without having confirmed its commit yet. The default retention window — three days — exists exactly to cover that margin: no reasonable Iceberg write should take more than three days to complete its commit, so any unreferenced file that's also more than three days old is, with very high confidence, really orphaned and not a write in progress.

This is the concrete reason remove_orphan_files is an engine operation and not a home-made script: it needs to precisely cross-reference the complete list of physical files against every commit in progress in the system's metadata, respecting a safety time window — work a plain os.walk() + set comparison, like this lesson's Step 1, deliberately doesn't attempt to solve safely for production.

Diagram: from rewritten metadata to bytes finally freed

flowchart LR
    L5["Lesson 5:\nexpire_snapshots()\n13 -> 2 snapshots"] --> M["Metadata: 2 tracked files\nDisk: 7 physical files"]
    M --> P1["This lesson's Step 1:\nos.walk() vs all_data_files()\n5 orphans, 9185 bytes"]
    P1 -.->|"(representative) -- Spark"| RM["remove_orphan_files\ndry_run first,\nretention >= 3 days"]
    RM -.-> F["Final disk state: 2 files\n(never executed in this environment)"]

Common mistakes

Deleting "by hand" a file this lesson's Step 1 flagged as orphaned. What happens: someone, with the list of five orphan files already identified, runs os.remove() directly on each one, thinking they already did the work equivalent to remove_orphan_files. Why it happens: the final result — the files disappear — looks identical. How to spot it: if your "cleanup" was based on a comparison taken at a single instant, with no retention window or verification against writes in progress, you ran exactly the risk the official warning describes — deleting a file another write still needed. How to fix it: use this lesson's Step 1 only to diagnose and understand the problem — exactly its stated purpose — never as a replacement for the official operation. In a real production environment, remove_orphan_files (Spark) is the only safe way to complete this cleanup, with its default three-day retention window respected.

Thinking remove_orphan_files is the same operation as expire_snapshots, just under another name. What happens: someone, after lesson 5, assumes running expire_snapshots() again, or with different parameters, would finish cleaning up the physical files. Why it happens: both operations have "cleanup" as their stated goal, so it's easy to assume they're interchangeable. How to spot it: check what each one tracks — expire_snapshots operates on the list of snapshots in the metadata; remove_orphan_files operates on the list of physical files in the filesystem, compared against the current metadata. How to fix it: they're two steps of the same maintenance flow, in order: first expire_snapshots (lesson 5, real in PyIceberg) stops referencing what's no longer needed; then remove_orphan_files (this lesson, representative in this environment) safely deletes what's left unreferenced — never the other way around, because remove_orphan_files without having expired first would find no candidates: everything would still be "tracked."

Exercises

Exercise 1 — Reproduce this lesson's Step 1 yourself, and confirm the two numbers. With lesson 5's state available, run the diagnostic script. Confirm 5 orphan files and 9185 total bytes.

See solution

If your table started from lesson 5's exact state, your output should match this lesson on both numbers — 5 files, 9185 bytes. The file names are going to be different (each includes a UUID generated at write time), but each one's size (1837 bytes) should match, because dim_product's four rows with this schema always produce the same Parquet file size.

Exercise 2 — Calculate what fraction of all the bytes this table ever wrote is still orphaned. Add up the two tracked files' size (use table.inspect.all_data_files().select(["file_size_in_bytes"])) and compare it against the 9185 orphan bytes.

See solution

The two tracked files — snap_v1's (V1, four rows) and the current one's (V2, four rows) — each weigh, roughly, the same as each of the five orphans (same schema, same row count): around 1837 bytes each, 3674 bytes total tracked. Against the 9185 orphan bytes, that means, right now, over 70% of all the bytes this table ever wrote correspond to files nobody needs anymore — a number that, in a production table with thousands of redundant writes instead of five, translates directly into real storage cost, the topic cost-optimization-caching-guide goes deeper on.

Exercise 3 — Explain, in your own words, why remove_orphan_files's default retention window is three days and not three seconds. Think about how long, in the worst case, a real distributed write can take to confirm its commit.

See solution

An Iceberg write on a real production table — unlike this guide's scripts, which run in under a second — can involve a distributed Spark job processing terabytes of data, with multiple tasks writing Parquet files in parallel for minutes or hours, before the final commit gets confirmed against the catalog. Throughout all that time, the files that write generates physically exist in the filesystem, but aren't yet referenced by any snapshot — the snapshot that's going to reference them only gets created at the end, when the commit is confirmed. If remove_orphan_files's retention window were three seconds, running that operation while a long write is in progress would delete files that write still needs, corrupting the final result. Three days is a deliberately generous margin, designed to cover even exceptionally long write jobs — the cost of being too conservative (orphan files that take a bit longer to clean up) is much lower than the cost of being too aggressive (a corrupted table).

Summary and next step

In this lesson you identified, with real code and no ambiguity, the five orphan files lesson 5's snapshot expiration left behind — 9,185 bytes, verified by comparing the filesystem against table.inspect.all_data_files(). You confirmed remove_orphan_files, just like lesson 4's compaction, isn't available in pure-Python PyIceberg 0.11.1, and documented its representative syntax against Iceberg's official documentation, including the real warning about the three-day retention window and why deleting "by hand" is dangerous.

Before moving on you should be able to: explain the difference between expire_snapshots (metadata) and remove_orphan_files (filesystem); and justify why a manual os.remove() never safely replaces the official operation.

Lesson 7 closes this module's maintenance loop with this whole guide's only Delta Lake mention: same problem — snapshots that accumulate cost, files that need compacting and cleaning up — completely different metadata mechanism.

Resources

  • Apache Iceberg — official documentation, "Maintenance," "Delete orphan files" section, source for the literal warning about the three-day retention window. iceberg.apache.org/docs/latest/maintenance. In English.
  • Apache Iceberg — official documentation, "Spark Procedures," remove_orphan_files section, exact source for this lesson's representative CALL syntax, including dry_run. iceberg.apache.org/docs/latest/spark-procedures. In English.
  • PyIceberg — API reference, the table.maintenance section, confirming once again only expire_snapshots is documented in this version. py.iceberg.apache.org/api. In English.
  • This same guide, module 2, lesson 6 — source of the discipline of exploring the warehouse/ on disk in a purely diagnostic way, without modifying anything. 06-inspecting-kioskos-table-on-disk.md. In Spanish.
  • This guide's DESIGN doc — module 7's section, "removing orphan files without losing the time travel you do need." src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.
← PreviousExpiring old snapshots, safelyNext →Delta Lake, by contrast