Module overview: from a Parquet file to a real Iceberg table

Why this module exists

If you've made it here having gone through the six previous guides in NIEVA's Data Engineering ecosystem, you've already seen Kiosko — the convenience-store chain with a delivery app, S01 Kiosko Centro in Bogotá, S02 Kiosko Norte in Lima, S03 Kiosko Sur in Santiago — solve the same underlying problem, over and over, with different tools. data-engineering-foundations-guide wrote its gold layer in Parquet, partitioned by date, with the overwrite-partition pattern: delete the old partition, write the new one. data-modeling-for-analytics-guide historized dim_product by hand, adding three columns (valid_from, valid_to, is_current) and writing its own MERGE INTO in DuckDB to maintain them. dbt-analytics-engineering-guide automated that exact same column technique with dbt snapshot, which generates dbt_valid_from, dbt_valid_to, and dbt_scd_id without anyone having to write the MERGE by hand. spark-and-distributed-processing-guide wrote fact_orders_at_scale.parquet — ten million rows — partitioned into folders with partitionBy("store_id"), the Hive pattern any big-data engine recognizes.

Notice something all four solutions have in common, even though they feel completely different: they all end in Parquet. Not one previous guide changed the file format — all of them, at some point, wrote a .parquet file to disk. What changed was what each team had to build around that file so it would behave like a real table: atomicity by hand (foundations), history columns by hand (data-modeling), automated history columns via an external tool (dbt), a folder convention you have to know by heart to take advantage of (spark). Four solutions, four different tools, and the exact same ceiling underneath all four: Parquet is a file format, not a table format, and a file format knows nothing about itself beyond its own rows — it doesn't know which version it is, it doesn't know what version existed before, it doesn't know whether the write that produced it finished completely or halfway.

This guide — lakehouse-and-iceberg-guide, the seventh of the ecosystem's 17 — teaches Apache Iceberg, the table format that moves that "behave like a table" responsibility off the engineer using it and onto the engine implementing it. This module 1 doesn't solve any of the four problems in full yet — that takes the rest of the guide — it does something more basic and more important: it names, one by one, with evidence and code quoted from each previous guide, the four exact times that same ceiling already showed up, and then takes the first real step toward the solution — installing PyIceberg and loading the same fact_orders.parquet as always as Kiosko's first Iceberg table.

The case that runs through the guide: Kiosko, without a single new data point

Kiosko doesn't change. The three stores are still the same (stores: store_id, store_name, city, inherited literally from foundations), the four products are still the same (P001 Bottled Water 600ml in beverages at 0.40, P002 Energy Bar in snacks at 0.60, P003 Instant Coffee Sachet in beverages at 0.35, P004 Phone Charger Cable in electronics at 2.10), and the fixed week of forty orders (2026-08-03 through 2026-08-09) still anchors the same total revenue that the six previous guides already verified, each with its own engine: 106.15 (S01=38.3, S02=38.8, S03=29.05). This guide doesn't invent a new case or a new model — it takes the fact_orders.parquet that spark-and-distributed-processing-guide (module 3 of that guide) already left written on disk, and turns it into a real Iceberg table.

The one genuinely new thing in this module is the Iceberg vocabulary added to what you already know: the catalog namespace is called kiosko, and the first table you're going to create is called kiosko.fact_orders — a two-part identifier (namespace.table) you're going to use for the rest of this guide.

An analogy: the box of loose photos and the album with an index

Imagine two ways of keeping the photos from an event. The first is a box of loose photos: each photo is well developed, on good-quality paper, perfectly sharp — but the box, as an object, knows nothing beyond containing photos. It doesn't know how many there are without someone counting them by hand. It doesn't know what order they were taken in unless someone numbered the backs. If someone pulls out ten photos and puts in ten different ones, the box has no way to tell you "this changed" — it simply contains whatever it contains right now, with no memory of what it contained yesterday.

The second way is an album with an index up front: the first page states, exactly, how many photos there are, in what order they're arranged, and on what page each one lives. If someone adds new photos, the album doesn't mix them in with the old ones without notice — it updates its index, and now that index points to a different collection, while the previous collection still exists, intact, in case someone wants to look at it again.

A loose .parquet file in a directory is the box of loose photos: the data inside is written perfectly well — columnar, compressed, typed, exactly what you've already used in the six previous guides — but the file, as an object, knows nothing about itself. It doesn't know if it's today's version or yesterday's. It doesn't know if the write that produced it finished completely. There's no index you can ask "what versions of this table existed, and in what order?" Apache Iceberg is the album: the same Parquet as always, exactly the same file format on the inside, but now with an index — the catalog and the chain of metadata — that knows, at all times, exactly which version of the table is current, and how to reach any earlier version without anyone having had to design that capability by hand.

Worked example: what a loose Parquet file can and can't tell you

Before installing anything from Iceberg, it's worth seeing, with code, exactly what a Parquet file does and doesn't give you on its own — the exact starting point of this module. You're going to reconstruct the fact_orders.parquet that spark-and-distributed-processing-guide left written in its module 3, using pyarrow directly (no cluster, no JVM), and read it like any other file:

# inspect_plain_parquet.py
import pyarrow.parquet as pq

# fact_orders.parquet: the same file Spark M3 already wrote,
# reconstructed here with pyarrow so this module is self-contained
pa_table = pq.read_table("fact_orders.parquet")

print(f"Rows: {pa_table.num_rows}")
print(f"Columns: {pa_table.num_columns}")
print()
print(pa_table.schema)

What to expect (verified by running the actual script, against Kiosko's forty rows):

Rows: 40
Columns: 7

order_id: string not null
store_id: string not null
product_id: string not null
quantity: int32 not null
unit_price: double not null
revenue: double not null
order_ts: timestamp[us] not null

This is everything the file, on its own, can tell you: how many rows it has right now, and with what schema. Notice everything it can't tell you, without someone building something extra outside the file itself:

• Is this the only version that ever existed, or is there a version from yesterday somewhere else? The file doesn't know — you'd need an external convention (a dated folder, a suffix in the name) even to start answering that.
• Did the write that produced this file finish completely? If the process that generated it died halfway through, there's no signal inside the Parquet itself to warn you — you'd simply have a truncated file, indistinguishable from a complete one until someone tries to read it carefully.
• Can I ask "show me what this table looked like on August 10", without having saved that version by hand myself? No. The file has no memory of earlier versions — overwriting it destroys the previous one, without a trace.

These three unanswered questions aren't a flaw in pyarrow, or in Parquet as a columnar format — Parquet is still, and will keep being throughout this whole guide, the file format that actually holds the data. They're, precisely, the exact definition of what a file is missing to be a table. Lesson 2 of this module names, one by one, the four times the whole Kiosko ecosystem already ran into this exact lack of answers, each time with a different, partial solution.

Diagram: what gets added on top of the same Parquet as always

flowchart TB
    subgraph YA["Already built (the six previous guides)"]
        A["fact_orders.parquet\n(columnar, typed, compressed)"]
        B["A loose file on disk\nwith no memory of earlier versions"]
    end

    subgraph NUEVO["This guide: Apache Iceberg"]
        C["Catalog: kiosko\n(points to current metadata)"]
        D["Metadata chain -> manifest list ->\nmanifest files (module 2)"]
        E["Automatic snapshots on every commit\n(module 3, no history columns)"]
        F["Safe schema evolution\n(module 4)"]
        G["Hidden partitioning and partition evolution\n(module 5)"]
        H["Native MERGE INTO / upsert\n(module 6)"]
    end

    A -->|"read, never rewritten"| C
    B -.->|"replaced by"| D
    C --> D --> E --> F --> G --> H

The map of this guide's 8 modules

lakehouse-and-iceberg-guide is the seventh guide in NIEVA's Data Engineering ecosystem. Eight modules make it up:

#	Module	What it's about
1	From file format to table format (you are here)	The four times Parquet alone wasn't enough; file vs. table; installing PyIceberg; Kiosko's first Iceberg table.
2	Anatomy of an Iceberg table	Catalog → metadata → manifest list → manifest files → data files; inspecting the table on disk and via the API.
3	Snapshots and time travel	Every write is a new snapshot; AS OF a snapshot-id; the P002 change recovered with zero history columns.
4	Schema evolution without rewriting	Why overwrite-partition was never atomic; adding/renaming/dropping columns without touching existing data.
5	Hidden partitioning and partition evolution	Hive folders vs. hidden partitioning; partition transforms; evolving the partition scheme without rewriting 10 million rows.
6	MERGE INTO and native upserts	The three ways Kiosko already solved the P002 change, plus the fourth: Iceberg's MERGE INTO via Spark and PyIceberg's table.upsert().
7	Catalogs, maintenance, and Delta Lake by contrast	Named production catalogs; compaction and snapshot expiration; Delta Lake mentioned exactly once.
8	Project: Kiosko's lakehouse	The capstone: all of Kiosko's tables on Iceberg, final verification, a map toward the sibling guides.

Notice the progression: this module 1 gives you the why and the first table. Module 2 gives you the anatomy — what's really inside the directory this module is going to create. Module 3 gives you the format's central superpower: traveling through time without having designed a single column for it. Modules 4, 5, and 6 give you the three operational guarantees a real lakehouse needs — schema, partitioning, upserts. And modules 7 and 8 close out with maintenance, a contrast with Delta Lake, and the capstone that ties every piece together.

The map of this module

Within module 1, eight lessons build the idea step by step:

Lesson    Question it answers
────────  ──────────────────────────────────────────────────────────────
L1        (this one) Where we're coming from, and where this module is going
L2        What, exactly, are the four times Parquet alone
          wasn't enough in this ecosystem?
L3        What's the exact difference between "file format"
          and "table format"?
L4        Install PyIceberg and create a local catalog, for real
L5        Create the kiosko namespace and the kiosko.fact_orders table
L6        Load fact_orders.parquet's 40 rows into Iceberg
L7        Verify the total is still 106.15, now read
          from an Iceberg table
L8        Project: Kiosko's first Iceberg table, end to end

Lesson 2 revisits, with code quoted literally from each previous guide, the four times you already saw this problem. Lesson 3 precisely defines what separates a file format from a table format, before installing a single tool. Lessons 4, 5, and 6 install PyIceberg, create the catalog, the namespace, and the table, and load Kiosko's real data. And lessons 7 and 8 verify, with the same number the six previous guides already confirmed (106.15), that the migration was exact.

The boundary: what does NOT belong in this module (or this guide)

This module installs local PyIceberg, with a SQL catalog backed by SQLite and filesystem storage — no cloud account, no JVM, no Spark. pip install "pyiceberg[sql-sqlite,pyarrow]" is the only install in this module.

And at the level of the whole guide, the boundary with the ecosystem's sibling guides is already drawn:

• Distributed computing in depth (partitioning and shuffle as the daily-work engine, Catalyst, caching) → spark-and-distributed-processing-guide. Spark appears exactly once, in module 6, as the SQL client that runs MERGE INTO against an Iceberg table.
• Conceptual dimensional modeling (why the grain is what it is, star vs. snowflake, SCD) → already taught by data-modeling-for-analytics-guide. Kiosko's star schema arrives already designed.
• Transformation as versioned code (dbt, ref(), declarative tests) → dbt-analytics-engineering-guide. There's no dbt project here: the dbt-iceberg adapter is named in module 7, without building a single model.
• Real orchestration (DAGs, sensors, managed retries) → airflow-and-declarative-orchestration-guide. All the code in this guide runs by hand, from the terminal.
• Real streaming/CDC → streaming-with-kafka-and-flink-guide. The P002 change still arrives as a fixed value declared in Python.
• Cloud-managed catalogs (AWS Glue Catalog, S3 Tables, Unity Catalog) → aws-core-services-guide. This guide uses 100% local catalogs.

Within this module 1 specifically: you're going to load the data exactly as Spark already left it partitioned — without touching the partitioning yet; turning that Hive partitioning into Iceberg's hidden partitioning is, precisely, module 5's job.

Common mistakes

Thinking Iceberg replaces Parquet as the file format. What happens: someone, on hearing "table format," assumes Iceberg is going to store the data in some new, proprietary binary format, different from Parquet. Why it happens: the name "table format" sounds, at first hearing, like a competing alternative to "file format." How to spot it: if after this module you expect to find .iceberg files instead of .parquet inside the table's directory, revisit this lesson's worked example — the data is still Parquet, exactly the same columnar format as always. How to fix it: Iceberg sits on top of Parquet, never replaces it — lesson 3 of this module draws that exact distinction, and the whole of module 2 shows you, on disk, that the data files are still .parquet, readable with any tool you already know.

Skipping lesson 2 because "I already lived through those four problems, I don't need them repeated to me." What happens: someone familiar with the six previous guides decides they can jump straight to installing PyIceberg without reading lesson 2. Why it happens: each individual problem already felt solved at the time, so revisiting them feels redundant. How to spot it: if you can't name, without looking back, which of the four problems each module of this guide (2 through 6) solves, you're missing lesson 2 — it's not decorative repetition, it's the map that connects each following module to a real, already-lived pain point. How to fix it: lesson 2 doesn't solve any problem again — it quotes, literally, the exact code from each previous guide, and names precisely what each solution was missing. That map is what makes installing Iceberg feel like a continuation, not like a new tool with no connection to what you've already learned.

Exercises

Exercise 1 — Name the unanswered question. Without looking at this lesson's worked example, write from memory the three questions a loose Parquet file can't answer on its own.

See solution

1. Is this the only version that existed, or is there an earlier version somewhere else? 2. Did the write that produced this file finish completely, or could it have stopped halfway? 3. Can I ask "show me what this table looked like on a past date," without having saved that version by hand myself? All three questions share a common root: a Parquet file only knows how to describe itself, right now — it has no memory of anything earlier, nor any way to guarantee its own write was atomic.

Exercise 2 — Trace the analogy yourself. Using the analogy of the box of loose photos and the album with an index, explain in 2-3 sentences what Iceberg's catalog represents in that analogy — even though you haven't installed it yet, based only on what you read in this lesson.

See solution

The catalog is the index at the front of the album: it doesn't contain the photos themselves — those are still the Parquet files, the actual content — but it knows, at all times, which photo collection is current and where to find its detailed index (the metadata file). When someone adds new photos, the catalog doesn't mix anything in blindly: it updates its pointer toward a new index, while the previous index — and the photos it pointed to — keeps existing, intact, for anyone who wants to look it up. Lesson 4 of this module installs that catalog for real, as a local SqlCatalog backed by SQLite.

Exercise 3 — Prediction. Before reading lesson 4, write your own hypothesis: why do you think PyIceberg needs a separate catalog, instead of simply reading and writing Parquet files directly, the way you already did in spark-and-distributed-processing-guide?

See solution

There's no single correct answer — this is a prediction exercise — but a reasonable hypothesis, based on this lesson's worked example, points to this: if Iceberg only read and wrote loose Parquet files, it would have the exact same problem you already saw — no file, however well written, knows which is "the current version" without something external telling it. The catalog is, precisely, that "something external": a single place, consulted on every operation, that tells any reader "the current version of kiosko.fact_orders is this one, it points to this exact metadata file." Without a catalog, every reader would have to guess which Parquet is current — exactly the problem Iceberg exists to solve.

Summary and next step

In this lesson you saw the exact point where the six previous guides in the ecosystem left off — four different solutions, all ending in Parquet, all building something extra around that file so it would behave like a table — and the three concrete questions a loose Parquet file can't answer on its own. You walked through the full map of this guide's eight modules and this module's eight lessons, and drew the boundary with the ecosystem's sibling guides.

Before moving on you should be able to: explain, in your own words, why the four solutions from the previous guides share the same ceiling even though they feel different; and name the three questions a loose Parquet file can't answer.

Lesson 2 earns the right to install any tool: before that, it revisits, one by one, with code quoted literally from each previous guide, the four exact times this ecosystem already ran into that ceiling.

Resources

• Apache Iceberg — official documentation, "What is Iceberg?", the formal table-format definition this lesson presents through the album analogy. iceberg.apache.org/docs/latest. In English.
• PyIceberg — official documentation (quickstart), the install and flow this module is going to install in lesson 4. py.iceberg.apache.org. In English.
• data-engineering-foundations-guide DESIGN doc — source of the overwrite-partition pattern lesson 2 revisits first. src/guides/data-engineering-foundations-guide/DISENO.md. In Spanish.
• data-modeling-for-analytics-guide DESIGN doc — source of dim_product_scd with valid_from/valid_to/is_current and the canonical P002 change. src/guides/data-modeling-for-analytics-guide/DISENO.md. In Spanish.
• dbt-analytics-engineering-guide DESIGN doc — source of dbt snapshot and its dbt_valid_from/dbt_valid_to/dbt_scd_id columns. src/guides/dbt-analytics-engineering-guide/DISENO.md. In Spanish.
• spark-and-distributed-processing-guide DESIGN doc — source of fact_orders.parquet and the Hive partitioning (partitionBy("store_id")) lesson 2 closes with. src/guides/spark-and-distributed-processing-guide/DISENO.md. In Spanish.
• This guide's DESIGN doc — the full map of the eight modules, including the market warning about Iceberg. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.