Module overview: the capstone, Kiosko's complete lakehouse

Why this module exists

The seven previous modules built, one at a time, each specific guarantee a table format adds to a loose Parquet file. Module 1 loaded kiosko.fact_orders as the first real Iceberg table. Module 2 mapped its complete anatomy — catalog, metadata, manifest list, manifest files, data files. Module 3 time-traveled over kiosko.dim_product and recovered P002's correct margin with no history column at all. Module 4 evolved kiosko.dim_store's schema, adding country without rewriting a single file. Module 5 partitioned kiosko.fact_orders_at_scale — ten million rows — hidden, and evolved that spec forward. Module 6 applied P002's change with MERGE INTO (representative, via Spark) and with table.upsert() (really executed). Module 7 named the production catalogs and pruned, with real evidence, the snapshots that no longer added anything.

Each of those seven modules lived in its own working directory, with its own freshly created kiosko_catalog.db, testing one guarantee at a time. None of the seven showed you the complete lakehouse: Kiosko's five tables — fact_orders, dim_store, dim_product, dim_date, fact_orders_at_scale — living together, in the same catalog, in the same namespace, ready to answer the same business question six different engines already answered in this ecosystem's six previous guides: how much did Kiosko sell this week (106.15), and what's P002's correct margin once its history gets reconstructed (10.8)?

This module is that assembly. It doesn't introduce any new Iceberg capability — every piece you're going to use, you already built, with your own hands, in a previous module. What it does is exactly what data-modeling-for-analytics-guide's module 8 and dbt-analytics-engineering-guide's module 8 did before: bring seven scattered pieces together into a single dimensional warehouse, verified end to end, with the same exact numbers those two guides already confirmed with their own engines.

The case that runs through the module: the complete lakehouse, not one more demo

Kiosko — the three-store convenience chain and its delivery app — has, by the time this module closes, five real Iceberg tables coexisting in the same kiosko namespace:

• kiosko.fact_orders: the fixed 40-order week, total revenue 106.15 (S01=38.3, S02=38.8, S03=29.05).
• kiosko.dim_store: the three stores, with country populated since its first commit (Bogotá→Colombia, Lima→Peru, Santiago→Chile) — the final result of the schema evolution module 4 already demonstrated step by step.
• kiosko.dim_product: four products, zero history columns, with P002's change (snacks/0.60 → health-snacks/0.68, effective 2026-08-15) applied with table.overwrite(), and the previous state recoverable with time travel — module 3's final result.
• kiosko.dim_date: August 2026's calendar, 31 rows, the piece no previous module had built yet — this capstone's only genuinely new table.
• kiosko.fact_orders_at_scale: ten million rows, partitioned by store_id and evolved with DayTransform over order_ts — module 5's final result.

The question this module answers isn't technical — it's a business question, the same one data-engineering-foundations-guide, python-for-data-engineering-guide, data-modeling-for-analytics-guide, dbt-analytics-engineering-guide, and spark-and-distributed-processing-guide already answered, each with its own engine: how much did Kiosko sell, and what's P002's real margin once its history gets reconstructed? The answer, over Iceberg, is still 106.15 in revenue and 10.8 in correct margin — and getting to that correct margin no longer needs a single valid_from column.

An analogy: the final scale model, with the five pieces already built

An architecture studio that spent seven weeks separately building a scale model's foundation, structure, electrical system, interior partitions, and landscaping arrives at the eighth week with every piece finished on the workbench — each tested separately, none joined to the others yet. The eighth week doesn't build any new piece: it assembles. It fits the foundation together with the structure, runs the electrical system through the already-built walls, places the interior partitions in their exact spot, and surrounds everything with the landscaping that, until then, lived in its own box. The result isn't technically more sophisticated than any of the seven individual pieces — it is, for the first time, a complete model, the only version a client can look at in one glance and understand as a single building.

This module is that eighth week. Every table you're going to assemble, you already built, tested and verified, in a previous module. What was missing was the table where the five pieces coexist together.

Diagram: where the five pieces come from

flowchart TB
    M1["Module 1:\nkiosko.fact_orders\n40 rows, 106.15"]
    M3["Module 3:\nkiosko.dim_product\nsnap_v1, time travel, 10.8/9.36"]
    M4["Module 4:\nkiosko.dim_store\n+country (schema evolution)"]
    M5["Module 5:\nkiosko.fact_orders_at_scale\npartitioned + evolved"]
    M6["Module 6:\ntable.upsert()\nMERGE INTO (representative)"]
    NEW["NEW in this module:\nkiosko.dim_date\n31 rows, August 2026"]

    M1 --> L3["L3: rebuilding\nthe star schema"]
    M4 --> L3
    NEW --> L3
    M3 --> L4["L4: reproducing\nP002's history with time travel"]
    L3 --> L4
    M5 --> L5["L5: evolving and\npartitioning at scale"]
    M6 --> L6["L6: merging updates\nthe native way"]
    L4 --> L6
    L5 --> L7["L7: what Kiosko\nstill needs"]
    L6 --> L7
    L7 --> L8["L8: closing project\nfor the WHOLE guide"]

The map of this module

Lesson    What it solves
────────  ──────────────────────────────────────────────────────────────
L1        (this one) Why this module exists, the five pieces, the map
L2        The brief -- why Kiosko needs ONE lakehouse, not seven
          scattered demos
L3        Assembles fact_orders, dim_store (with country), and dim_date --
          the three tables that do NOT change state -- as real
          Iceberg tables, in a single catalog
L4        dim_product with no history columns + the complete star
          joined with time travel: correct margin 10.8 vs broken 9.36
L5        fact_orders_at_scale: 10,000,000 rows partitioned,
          evolved, with the new franchise under the new spec
L6        P002's same change applied with table.upsert() --
          the native path -- verified byte for byte against L4
L7        Ecosystem close: the 7 sibling guides and what each one
          solves from what this lakehouse leaves pending
L8        Final project: the complete lakehouse, a single script,
          automated assert -- CLOSES the whole guide

What this module does NOT teach again

Every mechanism you're going to see run in lessons 3 through 6 was already taught, step by step, with its own analogy and its own common mistakes, in a previous module. This module doesn't repeat those explanations — it uses them. If any code step feels confusing, the exact reference to where it was first taught is flagged in each lesson: dim_store's schema evolution was explained in depth in module 4, table.overwrite()'s mechanics and time travel in module 3, PartitionSpec and its evolution in module 5, and table.upsert() in module 6. This capstone assumes you already know them, and focuses on something no previous module could show in isolation: what a real lakehouse's five tables look like, together, in the same catalog.

Common mistakes

Expecting this module to teach an Iceberg technique the previous seven didn't cover. What happens: someone arrives at this module expecting to learn something new about the format itself — an API they haven't seen before, an Iceberg capability not yet mentioned. Why it happens: "capstone" sounds, in some courses, like "the most advanced module," and it's easy to assume "advanced" means "technically new." How to spot it: if you finish lesson 3 without recognizing any PyIceberg API method you already used before, check: load_catalog(), create_table(), append(), overwrite(), scan(snapshot_id=...), update_spec(), upsert() — all come from the seven previous modules. How to fix it: this module is about integration, not new technique — the value is in seeing the five pieces coexist, verified against the same exact numbers data-modeling-for-analytics-guide and dbt-analytics-engineering-guide already confirmed with different engines.

Assuming kiosko.dim_date already existed in some previous module, and looking for it there. What happens: someone searches, across modules 1 through 7, for the code that creates kiosko.dim_date, and doesn't find it. Why it happens: this guide's DESIGN doc names dim_date since module 1's lesson 5 — as one of the five tables the kiosko namespace is going to contain "by the end of this guide" — and it's reasonable to assume "named" means "already built." How to spot it: if your search across previous modules finds no create_table("kiosko.dim_date", ...) at all, it isn't a mistake on your part — that table was deliberately left for this module. How to fix it: this module's lesson 3 is the first time kiosko.dim_date really gets created, with the exact same schema (date_key, calendar_date, day_of_week, month, quarter, year, is_weekend) data-modeling-for-analytics-guide already used for its own dim_date.

Exercises

Exercise 1 — Before reading lesson 3, predict which three of Kiosko's tables do NOT need to reproduce any evolution or time travel mechanism in this module, and why. Think about which of the lakehouse's five tables have a simple final state — a single version, no history to reconstruct — versus the ones that do.

See solution

kiosko.fact_orders, kiosko.dim_store, and kiosko.dim_date don't need to reproduce any evolution or time travel mechanism in this module: all three load in their final state, all at once. fact_orders never changed in any previous module — it was always the same 40 rows; dim_store already has country since its first commit in this module, because the mechanism of how that column gets added without rewriting data was already demonstrated, step by step, in module 4 — repeating it here would be reteaching, not assembling; and dim_date is a new table with no previous state to reconstruct. kiosko.dim_product, instead, does need to reproduce the complete mechanism (V1, overwrite(), snap_v1) because this whole capstone's central point is demonstrating the correct margin (10.8) is still recoverable with time travel inside the assembled lakehouse — and kiosko.fact_orders_at_scale needs to reproduce its partition evolution because two distinct spec_ids coexisting is, itself, part of what this module verifies.

Exercise 2 — Without looking at this module's map, list the seven sibling guides in the data-engineering-ecosystem you already know up to this point. Count how many you can name from memory before reading lesson 7.

See solution

Up to this point in the guide, these have already been explicitly named: data-engineering-foundations-guide (the non-atomic overwrite-partition, module 4), python-for-data-engineering-guide (production Python, named in the DESIGN doc), data-modeling-for-analytics-guide (the hand-historized dim_product, this whole guide's canonical thread), dbt-analytics-engineering-guide (dbt snapshot, compared in module 6), spark-and-distributed-processing-guide (the source Parquet and folder-based partitioning, module 5), and airflow-and-declarative-orchestration-guide (named for the boundary, without being implemented). This module's lesson 7 adds three more to this list with specific evidence from the complete capstone: streaming-with-kafka-and-flink-guide, data-reliability-and-governance-guide, aws-core-services-guide, advanced-sql-querying-guide, and cost-optimization-caching-guide — seven sibling guides in total, each with its own exact boundary.

Exercise 3 — Explain, in your own words, why this module's lesson 8 is called "project" and not "summary." In 2-3 sentences, justify why a capstone in this guide needs an executable script with assert, and wouldn't be enough with a table recalling the numbers already seen.

See solution

A summary would only repeat, in prose, numbers already seen in the seven previous modules — a claim with no new evidence. An executable project, instead, rebuilds the five tables from scratch, in a new directory, and verifies with assert that the result is still exactly the same: it doesn't "trust it's still 106.15," it tests it again, in the same "verify, don't trust" style every closing project in this guide — modules 1, 3, 4, 5, 6, and 7 — already applied. The difference between "summary" and "project" is, precisely, the difference between remembering a result and reproducing it.

Summary and next step

In this lesson you saw why this module exists: seven previous modules each built one specific Iceberg guarantee over one isolated Kiosko table — this module brings them together into a single lakehouse, with the five tables coexisting in the same catalog. You learned the full map of the eight lessons, and this capstone's only genuinely new piece: kiosko.dim_date, the calendar no previous module had built yet.

Before moving on you should be able to: name Kiosko's complete lakehouse's five tables and which module each one's mechanism comes from; and explain why this module doesn't teach any new Iceberg technique, but integrates the seven already learned.

Lesson 2 translates this assembly's need into a concrete brief: what Kiosko's management asks the data team, in its own words, and why "seven scattered demos" is no longer enough.

Resources

• PyIceberg — official documentation (quickstart), the same installation and catalog foundation supporting all eight modules of this guide. py.iceberg.apache.org. In English.
• Apache Iceberg — official documentation, reference version 1.11.0, the complete concept map this capstone assembles. iceberg.apache.org/docs/latest. In English.
• data-modeling-for-analytics-guide DESIGN doc — source of dim_date (schema date_key/calendar_date/day_of_week/month/quarter/year/is_weekend) and of P002's canonical change this capstone reproduces with time travel. src/guides/data-modeling-for-analytics-guide/DISENO.md. In Spanish.
• dbt-analytics-engineering-guide DESIGN doc — second independent confirmation of the same margin numbers (10.8/9.36) this capstone verifies. src/guides/dbt-analytics-engineering-guide/DISENO.md. In Spanish.
• This guide's DESIGN doc — the full map of the eight modules, including this module 8's complete section. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.