Compacting small files

Description

Lesson 3 measured an old snapshots problem: redundant writes nobody needs anymore, but that still take up space until someone expires them. This lesson measures a different problem, one that can exist even in a table with no redundant snapshot at all: small files inside the current snapshot. You're going to load Kiosko's same 40-order week as always, but day by day instead of in a single append() — exactly how it would arrive in a real pipeline that receives one file per day — and you're going to confirm that alone fragments the table into more files than necessary.

Connection to the module. This lesson doesn't touch kiosko.dim_product — it uses a new table, kiosko.fact_orders_daily_batches, following the same dedicated-table discipline you already saw in module 6 (dim_product_upsert_demo). The reason: this lesson's problem is orthogonal to lesson 3's — none of the seven writes you're going to make here is redundant, each one brings real, new business data, and yet the result is a fragmented table.

Why this lesson uses fact_orders, not dim_product

kiosko.dim_product has four rows — no matter how many times you rewrite it, every overwrite() always fits in a single small Parquet file. The "small files" problem doesn't show up there naturally. kiosko.fact_orders, instead, is the kind of table where this problem shows up all the time in production: it receives data at some frequency — once a day, once an hour, sometimes in streaming — and each arrival typically becomes its own file. This lesson rebuilds that situation with the same data as always, grouped by the real day each order happened.

An analogy: the same album, now with one photo per visit instead of a full roll

The supermarket shelf, in modules 1 and 3, always got restocked completely, so every photo captured the whole shelf. Imagine, instead, a customer who comes in seven separate times during the week, and the quality-control employee takes a new photo every time that customer leaves, even if they only moved two or three products. By the end of the week you have seven partial photos, each one valid and necessary — none is redundant, each documents something real that happened — but putting them together to answer "what does the whole shelf look like today?" requires looking at all seven, one by one, instead of a single complete photo. Compacting, in this context, isn't "deleting photos" — that would lose information — it's reprinting the same content on fewer pages: the same information, organized in a way that's more efficient to read.

Worked example: seven daily arrivals, seven live files

Step 1 — Load Kiosko's week, one append() per day

# fact_orders_daily_batches.py -- the same 40-order week, loaded day by day
import os
from datetime import datetime

import pyarrow as pa
from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import DoubleType, IntegerType, NestedField, StringType, TimestampType

warehouse_path = os.path.abspath("kiosko_warehouse")
catalog_db_path = os.path.abspath("kiosko_catalog.db")

catalog = load_catalog(
    "kiosko", type="sql",
    uri=f"sqlite:///{catalog_db_path}", warehouse=f"file://{warehouse_path}",
)

# RAW_ORDERS_BY_DAY -- the same 40 rows from module 1, lesson 6, grouped by
# the real day they happened (2026-08-03 .. 2026-08-09, 7 days, 8+6+2+5+7+9+3=40)
from raw_orders_by_day import RAW_ORDERS_BY_DAY

FACT_ORDERS_SCHEMA = Schema(
    NestedField(field_id=1, name="order_id", field_type=StringType(), required=True),
    NestedField(field_id=2, name="store_id", field_type=StringType(), required=True),
    NestedField(field_id=3, name="product_id", field_type=StringType(), required=True),
    NestedField(field_id=4, name="quantity", field_type=IntegerType(), required=True),
    NestedField(field_id=5, name="unit_price", field_type=DoubleType(), required=True),
    NestedField(field_id=6, name="revenue", field_type=DoubleType(), required=True),
    NestedField(field_id=7, name="order_ts", field_type=TimestampType(), required=True),
)
pa_schema = pa.schema([
    pa.field("order_id", pa.string(), nullable=False),
    pa.field("store_id", pa.string(), nullable=False),
    pa.field("product_id", pa.string(), nullable=False),
    pa.field("quantity", pa.int32(), nullable=False),
    pa.field("unit_price", pa.float64(), nullable=False),
    pa.field("revenue", pa.float64(), nullable=False),
    pa.field("order_ts", pa.timestamp("us"), nullable=False),
])

table = catalog.create_table("kiosko.fact_orders_daily_batches", schema=FACT_ORDERS_SCHEMA)

for day, orders in RAW_ORDERS_BY_DAY:
    rows = []
    for order_id, store_id, product_id, quantity, unit_price, ts in orders:
        rows.append({
            "order_id": order_id, "store_id": store_id, "product_id": product_id,
            "quantity": quantity, "unit_price": unit_price,
            "revenue": round(quantity * unit_price, 10),
            "order_ts": datetime.fromisoformat(ts),
        })
    table.append(pa.Table.from_pylist(rows, schema=pa_schema))
    print(f"append() for {day}: {len(rows)} rows")

What to expect (verified by running the real script):

append() for 2026-08-03: 8 rows
append() for 2026-08-04: 6 rows
append() for 2026-08-05: 2 rows
append() for 2026-08-06: 5 rows
append() for 2026-08-07: 7 rows
append() for 2026-08-08: 9 rows
append() for 2026-08-09: 3 rows

Seven append() calls, each one with a single day's real orders — nothing redundant, every row is genuine business information, exactly like it would arrive from a real transactional system exporting one file per day.

Step 2 — Confirm the total, and count the live files

total_rows = table.scan().to_arrow().num_rows
fact_rows = table.scan().to_arrow().to_pylist()
total_revenue = round(sum(r["revenue"] for r in fact_rows), 2)
print(f"\ntotal rows: {total_rows}, total revenue: {total_revenue}")

live_files = table.inspect.files()
print("table.inspect.files().num_rows:", live_files.num_rows)
for row in live_files.select(["file_path", "record_count", "file_size_in_bytes"]).to_pylist():
    print(" ", row["file_path"].split("/")[-1], "record_count=", row["record_count"],
          "bytes=", row["file_size_in_bytes"])

What to expect (verified by running the real script; the file names are from your own run, different every time — record_count and file_size_in_bytes are deterministic):

total rows: 40, total revenue: 106.15

table.inspect.files().num_rows: 7
  <file-1>.parquet record_count= 3 bytes= 2777
  <file-2>.parquet record_count= 9 bytes= 2928
  <file-3>.parquet record_count= 7 bytes= 2873
  <file-4>.parquet record_count= 5 bytes= 2834
  <file-5>.parquet record_count= 2 bytes= 2747
  <file-6>.parquet record_count= 6 bytes= 2857
  <file-7>.parquet record_count= 8 bytes= 2873

The total revenue — 106.15 — matches, once again, the number the six previous guides in the ecosystem already verified: loading the data day by day instead of in a single append() didn't change a single cent of the business result. What did change is the table's physical shape: seven Parquet files, all live, all needed to answer a query over the complete table — none is a candidate for expire_snapshots (lesson 5's technique doesn't apply here, because there's no redundant snapshot to expire) — and each one tiny: between 2 and 9 rows, between 2.7 and 2.9 KB.

Why this is a problem, even though no file is redundant

Apache Iceberg's official documentation sums it up like this: "More data files leads to more metadata stored in manifest files, and small data files causes an unnecessary amount of metadata and less efficient queries from file open costs." Scanning kiosko.fact_orders_daily_batches completely — even a trivial SELECT * — forces the query engine to open seven separate files instead of just one. With seven files of a few KB each, the relative cost of opening each file (locating it in storage, reading its Parquet footer, planning the read) can far exceed the cost of reading the data itself — exactly why a 500 MB file almost always reads more efficiently than a hundred 5 MB files, even though the total data volume is identical.

Compacting is the operation that solves this: combining several small files into fewer, larger ones, without changing a single row of content — Iceberg's default strategy for this is called bin-packing: it groups small files until they approach a target size (target-file-size-bytes, 512 MB by default in Iceberg's write configuration), and writes that group as a single new file.

The real operation: rewrite_data_files — not available in pure-Python PyIceberg 0.11.1

It was verified, against PyIceberg 0.11.1's installed source code and against its official API reference, that no method equivalent to compaction exists on Table or on table.maintenance — the only documented operation under table.maintenance in this version is expire_snapshots() (lesson 5). Data file compaction, in the 2026 Iceberg ecosystem, is an operation that runs in parallel over a distributed compute engine — typically Spark, via the rewriteDataFiles action or the rewrite_data_files SQL procedure — and no pure Python client implements it yet.

This is exactly the pattern module 6, lesson 3 already documented with MERGE INTO: the syntax exists, it's verified against the official documentation, but it doesn't run in this environment. The following block is marked as representative:

-- (representative) -- syntax verified against Iceberg's official documentation,
-- NOT executed in this environment: PyIceberg 0.11.1 doesn't implement rewrite_data_files.

-- compact kiosko.fact_orders_daily_batches with the default bin-pack strategy
CALL local.system.rewrite_data_files('kiosko.fact_orders_daily_batches');

-- explicit version, setting a smaller target size (Iceberg's defaults
-- are designed for real production tables, not 40 lab rows)
CALL local.system.rewrite_data_files(
    table => 'kiosko.fact_orders_daily_batches',
    options => map('target-file-size-bytes', '134217728')  -- 128 MB
);

What to expect (representative): the seven files of between 2.7 and 2.9 KB would combine into one single Parquet file with the same 40 rows — the same total revenue (106.15), the same structure, no data lost or duplicated — and table.inspect.files().num_rows would go from 7 to 1. The procedure, run on Spark, also produces a structured report (rewritten_data_files_count, added_data_files_count, rewritten_bytes_count) confirming exactly how many files went in and how many came out — the same "verify, don't assume" discipline this guide applied in every previous module, now applied to an operation that wasn't executed in this environment.

Going deeper: compacting produces a new snapshot, it doesn't avoid one

It's worth a precise clarification that often gets misunderstood: compacting does not reduce the number of snapshots — on the contrary, it adds one more. The operation type Iceberg records for a compaction is replace (distinct from append, overwrite, or delete): Iceberg's formal specification describes it as "Data and delete files were added and removed without changing table data" — the table's logical content doesn't change by a single row, but which physical files represent that content does change, and that, like any change, gets archived as a new snapshot. This is exactly why lesson 5 — expire_snapshots — and this lesson solve related but distinct problems: compacting cleans up the "too many small files in the current snapshot" problem; expiring cleans up the "too many old snapshots nobody needs anymore" problem — and, without meaning to, a rewrite_data_files with no expire_snapshots run afterward would leave the seven original files still tracked by the pre-compaction snapshot, not freeing a single byte until that snapshot also expires.

Diagram: seven live files, one compaction candidate

flowchart LR
    D1["2026-08-03\n8 rows"] --> F["kiosko.fact_orders_daily_batches\ncurrent snapshot"]
    D2["2026-08-04\n6 rows"] --> F
    D3["2026-08-05\n2 rows"] --> F
    D4["2026-08-06\n5 rows"] --> F
    D5["2026-08-07\n7 rows"] --> F
    D6["2026-08-08\n9 rows"] --> F
    D7["2026-08-09\n3 rows"] --> F

    F -->|"7 live files,\nnone redundant"| Q["Cost: 7 file opens\nper complete query"]
    Q -.->|"rewrite_data_files\n(representative)"| C["1 compacted file\nsame 40 rows\n+1 snapshot operation=replace"]

Common mistakes

Confusing this lesson's problem with lesson 3's. What happens: someone, after seeing "7 files" in this lesson and "7 files" in lesson 3 (a numeric coincidence, not a causal relationship), assumes expire_snapshots would also solve this lesson's small-files problem. Why it happens: both numbers are "7," and both are about Parquet files, so it's easy to mix them up. How to spot it: check whether the files in question are all live (like in this lesson, none redundant) or whether some are redundant (like in lesson 3, six of seven with no additional business value). How to fix it: expire_snapshots (lesson 5) solves files nobody needs anymore, left behind by old snapshots; rewrite_data_files/compaction (this lesson) solves files that are needed, but are fragmented into more pieces than convenient. They're different axes, and a real table can have both problems at the same time, as would be the case if lesson 3's nightly pipeline had also arrived in small daily batches.

Assuming a smaller target-file-size-bytes is always better because "it produces more manageable files." What happens: someone, seeing this lesson's 128 MB example, concludes smaller files are, in general, safer or easier to work with. Why it happens: in a 40-row lab dataset, any target size above a few KB produces the same result — a single file — so the "smaller is more manageable" intuition never gets tested. How to spot it: if your real table has millions of rows and you set too small a target-file-size-bytes, you're going to end up with many "compacted" files that are still, relatively, small — the same file-open-cost problem this lesson describes, just after having spent the work of compacting. How to fix it: Iceberg's default value (512 MB, the same constant that shows up in the table's write configuration) is a reasonable starting point for real-scale datasets — adjusting it down only makes sense if your query pattern filters aggressively by partition and you prefer smaller files per partition, a topic cost-optimization-caching-guide goes deeper on, not this lesson.

Exercises

Exercise 1 — Reproduce the full experiment yourself, and confirm the two numbers. In a new directory, run this lesson's script. Confirm 40 total rows, 106.15 in revenue, and 7 live files.

See solution

If your environment has PyIceberg 0.11.1 installed, your output should match this lesson on all three numbers. The file names (file_path) are going to be different — each includes a UUID generated at write time — but record_count (3, 9, 7, 5, 2, 6, 8, in some order) and file_size_in_bytes should match, because they depend only on each file's content, not on when you ran the script.

Exercise 2 — Calculate how many files this same week would have produced if the pipeline had grouped by hour instead of by day. Using the timestamps from raw_orders_by_day (inherited from module 1, lesson 6), count how many distinct hours (order_ts truncated to the hour) have at least one order.

See solution

Counting the unique hourly timestamps across Kiosko's 40 orders, the result is well over seven distinct hours across the week — each day has between 2 and 4 distinct hours with at least one order — so grouping by hour instead of by day would produce more than twenty files, each with an even smaller handful of rows than this lesson's. This exercise illustrates the direct relationship between write frequency and fragmentation: the more often you write, with no compaction mechanism running behind it, the more small files you accumulate — the exact reason streaming-fed tables (outside this guide's scope, streaming-with-kafka-and-flink-guide's territory) almost always need scheduled compaction as a normal part of their operation, not an exception.

Exercise 3 — Explain, in your own words, why rewrite_data_files produces a replace-type operation and not overwrite. Think about the difference between "changing a table's logical content" and "changing how that same content is physically organized."

See solution

overwrite() (module 3) changes the logical content: the rows the table returns before and after the operation are different — P002 goes from snacks to health-snacks, for example. replace, the operation type a compaction produces, doesn't change a single row of what the table returns — the same 40 orders, with the same values, are still there — the only thing that changes is how many physical files represent that same content. Marking this operation with a distinct type (replace, not overwrite) lets any tool reading the table's history — including expire_snapshots's own internal logic — unambiguously distinguish between "here something a business user needs to know about changed" and "here the files just got reorganized, the content is identical to the previous snapshot's."

Summary and next step

In this lesson you identified a maintenance problem distinct from lesson 3's: not redundant snapshots, but small files inside a completely legitimate current snapshot. You loaded kiosko.fact_orders_daily_batches day by day — seven real append()s, none redundant — and confirmed, with table.inspect.files(), the result is seven tiny files, between 2.7 and 2.9 KB each. You verified PyIceberg 0.11.1 doesn't implement rewrite_data_files in pure Python, and documented Spark's representative syntax that does solve it, including the precision that compacting produces a new snapshot (operation=replace), it doesn't avoid one.

Before moving on you should be able to: tell this lesson's problem apart from lesson 3's; and explain why compacting without later expiring leaves the old files' space unreleased.

Lesson 5 returns to kiosko.dim_product and lesson 3's thirteen snapshots, and this time it really runs: table.maintenance.expire_snapshots(), with snap_v1 explicitly protected.

Resources

• Apache Iceberg — official documentation, "Maintenance," "Compact data files" section, source for the exact quote about small files' cost. iceberg.apache.org/docs/latest/maintenance. In English.
• Apache Iceberg — official documentation, "Spark Procedures," rewrite_data_files section, exact source for this lesson's representative CALL syntax. iceberg.apache.org/docs/latest/spark-procedures. In English.
• PyIceberg — API reference, the table.maintenance section, confirming only expire_snapshots is documented in this version. py.iceberg.apache.org/api. In English.
• This guide's DESIGN doc — module 7's section, "small file compaction." src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.