The brief: Kiosko needs a lakehouse, not seven demos

Description

Kiosko's management — the same operations manager who, in data-engineering-foundations-guide, asked for a weekly report by hand, and who, in data-modeling-for-analytics-guide, asked for "a single warehouse, not seven demos" — now has a different question, aimed specifically at the team that spent seven modules learning Apache Iceberg: "I already know each table, separately, works — but do I actually have a lakehouse, or do I have seven Iceberg experiments that never shared a catalog?" This lesson turns that question into a concrete brief, with the same verifiable requirements data-modeling-for-analytics-guide's and dbt-analytics-engineering-guide's capstones already used, adapted to what Iceberg specifically adds: no history columns at all, everything verifiable with time travel.

Connection to the module. This lesson doesn't build any table yet — that starts in lesson 3. Its job is translating management's question into a list of requirements, each mapped to the lesson in this module that solves it, and confirming, with evidence quoted from the seven previous modules, that the material already exists.

An analogy: the occupancy certificate, not a tour of separate job sites

A building inspector who visits, one by one, seven different job sites from the same contractor — one building's foundation, another's structure, a third's electrical system — can confirm, with complete honesty, that all seven are well built. But that inspector can't sign an occupancy certificate until the seven pieces are, physically, the same building: the same foundation holding up the same structure, the same electrical system running through the same walls. The occupancy certificate doesn't evaluate each piece's quality again — that's already done — it evaluates whether, together, they form something someone can live in. This module is that certificate: it doesn't re-prove table.overwrite() works, or that PartitionSpec evolves without rewriting data — modules 3 and 5 already proved that — it proves the five tables, together, in the same catalog, form a habitable lakehouse.

The brief, in management's own words

"Every module proved something real to me: that Iceberg remembers the past with nobody writing valid_from, that I can add a column without breaking anything, that a ten-million-row table can be partitioned and evolved without rewriting the whole thing. All of that convinced me. But when I asked my team 'so, do I have a lakehouse now?', the answer was 'not yet — every demo lives in its own catalog, created from scratch, and none of them knows the others exist.' I need one concrete thing: a single catalog, with Kiosko's five tables coexisting there, and confirmation that if I ask 'how much did we sell, with the energy bar's correct margin,' the answer is still the same one six different teams already gave me before: 106.15 in revenue, 10.8 in margin."

Four concrete requirements come out of that paragraph, and each maps to a piece you already built in a previous module:

Management's requirement	Piece you already built	Module
"a single catalog, with the five tables coexisting"	every table lived in its own kiosko_catalog.db, created from scratch per lesson	Modules 1-7, assembled here
"remembering the past with no valid_from"	dim_product with no history columns + time travel	Module 3
"partitioning and evolving without rewriting"	fact_orders_at_scale, PartitionSpec, update_spec()	Module 5
"the same 106.15/10.8 six teams already confirmed"	Kiosko's canonical revenue and margin, verified with different engines in every guide	The whole ecosystem

The first row is what this module really adds: no previous module served the five tables in the same catalog, in the same run. That's exactly what this module's lessons 3 through 6 build.

Worked example: confirming what material you have before assembling

Before promising management anything, confirm the previous modules' seven pieces are available, with the exact numbers you already know, so this module's assembly doesn't depend on memory but on evidence.

# confirm_material.py -- no real catalog yet, just a verifiable inventory
MODULE_DELIVERABLES = [
    ("Module 1", "kiosko.fact_orders loaded",             "40 rows, revenue 106.15"),
    ("Module 2", "table anatomy mapped",                  "catalog -> metadata -> manifest list -> manifest files -> data files"),
    ("Module 3", "dim_product with no history + time travel", "P002: snacks/10.8 (correct) vs health-snacks/9.36 (broken)"),
    ("Module 4", "dim_store evolved",                     "country: S01=Colombia, S02=Peru, S03=Chile"),
    ("Module 5", "fact_orders_at_scale partitioned",       "10,000,000 rows, S01=9,575,000.00, spec evolved"),
    ("Module 6", "MERGE INTO + table.upsert()",            "rows_updated=1, rows_inserted=0, same result as overwrite()"),
    ("Module 7", "catalogs + maintenance",                 "real expire_snapshots(), 4 production catalogs named"),
]

print("=== Kiosko: confirming the 7 modules' material, before assembling the lakehouse ===\n")
for module, deliverable, evidence in MODULE_DELIVERABLES:
    print(f"{module:10} {deliverable:38} {evidence}")

print("\nWhat's really missing: NO new Iceberg capability -- just assembling the 5 tables")
print("in the same catalog. This module's only genuinely new table: kiosko.dim_date.")

What to expect. Running python3 confirm_material.py, the output is exactly this:

=== Kiosko: confirming the 7 modules' material, before assembling the lakehouse ===

Module 1   kiosko.fact_orders loaded               40 rows, revenue 106.15
Module 2   table anatomy mapped                    catalog -> metadata -> manifest list -> manifest files -> data files
Module 3   dim_product with no history + time travel  P002: snacks/10.8 (correct) vs health-snacks/9.36 (broken)
Module 4   dim_store evolved                       country: S01=Colombia, S02=Peru, S03=Chile
Module 5   fact_orders_at_scale partitioned        10,000,000 rows, S01=9,575,000.00, spec evolved
Module 6   MERGE INTO + table.upsert()             rows_updated=1, rows_inserted=0, same result as overwrite()
Module 7   catalogs + maintenance                  real expire_snapshots(), 4 production catalogs named

What's really missing: NO new Iceberg capability -- just assembling the 5 tables
in the same catalog. This module's only genuinely new table: kiosko.dim_date.

Seven rows, seven concrete numerical pieces of evidence, none invented again here — each one is, literally, its own module's closing project's already-verified result. Notice the pattern: this brief doesn't ask for "building something new" in the sense of an Iceberg capability — it asks for bringing together what already exists, in a single catalog, so the business question ("how much did Kiosko sell, with the correct margin?") has one answer, not seven answers scattered across seven different directories.

Diagram: from management's question to the technical brief

flowchart TD
    A["'Every demo convinced me\nseparately, but I need\nONE lakehouse, not seven catalogs'"] --> B["Technical brief (this lesson)"]
    B --> C["A single catalog, 5 tables\ncoexisting -- L3, L4, L5"]
    B --> D["Remembering the past with no\nvalid_from -- dim_product,\ntime travel (L4)"]
    B --> E["Partitioning/evolving without\nrewriting -- at_scale (L5)"]
    B --> F["The same 106.15/10.8\n6 engines already confirmed"]
    C --> G["Assembled lakehouse,\nverified with assert (L8)"]
    D --> G
    E --> G
    F --> G

Going deeper: why "a single catalog" is the requirement no previous module could meet

Notice something deliberate about the seven previous modules: each one, in its lesson 4 or 5, ran exactly the same startup pattern — warehouse_path = os.path.abspath("kiosko_warehouse"), catalog_db_path = os.path.abspath("kiosko_catalog.db"), load_catalog("kiosko", type="sql", uri=..., warehouse=...). That pattern, repeated seven times, in seven different working directories, is exactly correct for each module's pedagogical purpose: isolating one guarantee at a time, with no module's state interfering with the next. But it has a direct consequence nobody mentioned until now: the seven kiosko_catalog.dbs you generated, one per module, are seven physically distinct catalogs, each with its own kiosko.fact_orders table, its own kiosko.dim_product, with no relationship to each other.

That is, precisely, what management detected in their quote: "every demo lives in its own catalog, created from scratch." This module solves that fragmentation the only honest way possible — not by merging the seven previous catalogs (an operation Iceberg doesn't offer, and that wouldn't make pedagogical sense either), but by rebuilding the five final tables, once, in a single new catalog, exactly like data-modeling-for-analytics-guide's and dbt-analytics-engineering-guide's capstones did before. Lesson 3 opens that single catalog; lessons 4, 5, and 6 keep writing to it, without recreating a new one per lesson — the first time in this whole guide several consecutive lessons deliberately share the same kiosko_warehouse/.

Common mistakes

Interpreting the brief as a request for a sixth table or a new Iceberg capability. What happens: someone, on reading "I need a real lakehouse," assumes the brief asks for researching something the seven previous modules didn't cover — a new partition transform type, an additional maintenance operation. Why it happens: "real lakehouse" sounds like something bigger than what's already built, and it's easy to confuse "bigger" with "more techniques." How to spot it: if your plan for this module includes learning a PyIceberg API method that doesn't appear in any previous module, you've drifted from the brief — reread this lesson's table, the seven rows already cover everything the business asked for. How to fix it: the brief asks for assembly in a shared catalog, not invention — management's four requirements get solved by rebuilding, once, the tables you already built, not by designing an eighth technique.

Assuming "a single catalog" means merging the seven already-existing catalogs from previous modules. What happens: someone looks, in PyIceberg's API, for a method to combine or import tables from an existing SqlCatalog into another. Why it happens: "a single catalog with the five tables" sounds, on first read, like a catalog-migration operation. How to spot it: if your search through PyIceberg's documentation finds no "catalog merging" method (because it doesn't exist, and doesn't need to), you're looking for the wrong solution. How to fix it: lesson 3 merges nothing — it rebuilds the tables from Kiosko's same fixed data (RAW_ORDERS, DIM_STORE_ROWS, etc.) in a new catalog, exactly like every previous module's closing project already did with its own individual table.

Treating the brief's four requirements as independent from each other. What happens: someone builds the five tables in the same catalog, but never joins them in a single query confirming P002's correct margin against the complete revenue. Why it happens: each requirement in the brief mentions a different capability (shared catalog, time travel, partitioning, verification), so treating them as four separate deliverables seems natural. How to spot it: if your final lakehouse has the five tables loaded but no query joining them — fact_orders + dim_store + dim_date + dim_product AS OF snap_v1 — you're missing the piece that really proves it's a lakehouse, not just five parallel tables. How to fix it: this module's lesson 4 is precisely that join — the same kind of conceptual point-in-time JOIN data-modeling-for-analytics-guide already taught, now solved with table.scan(snapshot_id=snap_v1) instead of a BETWEEN valid_from AND valid_to.

Exercises

Exercise 1 — Explain why the seven kiosko_catalog.dbs from modules 1 through 7 can't simply get "copied" into a common directory to form this module's lakehouse. Think about what would happen if two of those seven catalogs, by chance, each had a table named kiosko.dim_product with different content.

See solution

Copying the seven kiosko_catalog.dbs into the same directory wouldn't produce a single lakehouse — it would produce seven SQLite files with the same name, overwriting each other, or seven catalogs with the same kiosko namespace but conflicting content. The concrete case that illustrates the problem: module 3 left kiosko.dim_product with two snapshots (V1, V2 via overwrite()); module 7 rebuilt a different version of that same table, with five additional nights of redundant writes, in its own isolated catalog. If both catalogs coexisted under the same name, there'd be no deterministic way to decide which kiosko.dim_product is "the real one." The only clean solution — the one this module applies — is rebuilding the final tables, once, in a new catalog, from the same fixed data you already know, not trying to merge write histories that were never meant to coexist.

Exercise 2 — Find the implicit reproducibility requirement in management's quote. Reread the complete quote. Without looking at "The brief"'s table, identify an implicit requirement not in the table, related to numbers' reliability over time.

See solution

A reasonable answer: the phrase "the answer is still the same one six different teams already gave me before" implies the Iceberg lakehouse's result has to match, number for number, guides that used completely different engines — DuckDB in data-modeling-for-analytics-guide, dbt+DuckDB in dbt-analytics-engineering-guide, Spark in spark-and-distributed-processing-guide. That implicit requirement — "the storage engine shouldn't change the business result" — is the same reproducibility discipline holding up this entire guide: Kiosko's data is always fixed, never generated with random or datetime.now(), precisely so comparing 106.15 and 10.8 across seven different guides makes sense.

Exercise 3 — Argue why "a single catalog with the five tables" is a different requirement from "five correct tables." In 2-3 sentences, explain what Kiosko gains by having the five tables in the same catalog that it wouldn't gain with five separate catalogs, each with one correct table.

See solution

Five correct tables in five separate catalogs would demonstrate every Iceberg mechanism works — exactly what modules 1 through 7 already proved — but they wouldn't let you answer any question that needs more than one table at once: calculating margin by category requires joining dim_product with fact_orders; filtering by country requires joining dim_store with fact_orders. A single catalog, with the five tables coexisting under the same kiosko namespace, is what turns five correct pieces into a queryable lakehouse — the exact difference between having the ingredients and having the recipe already plated on a single dish.

Summary and next step

In this lesson you translated management's question — "do I already have a lakehouse, or seven Iceberg demos?" — into a brief with four concrete requirements, each mapped to a piece you already built in the seven previous modules. You confirmed, with confirm_material.py, that the seven pieces are ready, with their exact numbers. And you saw why this module's central requirement — a single catalog, with the five tables coexisting — is something no previous module, by pedagogical design, could meet yet.

Before moving on you should be able to: repeat, from memory, the brief's four requirements and which module each one corresponds to; and explain why "a single catalog" isn't the same as "five separately correct tables."

Lesson 3 starts building: it opens this module's single catalog and loads the three tables that don't change state — fact_orders, dim_store with country, and the new dim_date — the complete lakehouse's first third.

Resources

• data-modeling-for-analytics-guide DESIGN doc — the same brief pattern ("a single warehouse, not seven demos") this lesson adapts to Iceberg. src/guides/data-modeling-for-analytics-guide/DISENO.md. In Spanish.
• dbt-analytics-engineering-guide DESIGN doc — second confirmation of the same capstone pattern, applied to a complete dbt project. src/guides/dbt-analytics-engineering-guide/DISENO.md. In Spanish.
• PyIceberg — official documentation (quickstart), the foundation for load_catalog() this module reuses on a single catalog starting in lesson 3. py.iceberg.apache.org. In English.
• This guide's DESIGN doc — the full map of the eight modules, including this module 8's complete section. src/guides/lakehouse-and-iceberg-guide/DISENO.md. In Spanish.