What Kiosko still needs

Description

The lakehouse that exists when lesson 6 closes is real: five Iceberg tables — fact_orders, dim_store, dim_product, dim_date, fact_orders_at_scale — coexisting in the same catalog, with real ACID, time travel with no history columns, hidden and evolved partitioning, and two native write paths converging on the same result. And, at the same time, it's still a lab lakehouse in one very concrete sense: it runs on your laptop, on a kiosko_catalog.db file a few megabytes in size, with every operation triggered by hand from your own terminal, over a P002 change that's still a fixed value written in Python. This lesson isn't a criticism of what you built — it's the honest map of the seven exact boundaries this guide, from its own design, decided not to cross, and the precise name of the sibling guide that does cross each one, each with concrete evidence from this very lakehouse as its starting point.

Connection to the module. This guide's DESIGN doc named these seven boundaries from its first line, in the "does NOT belong in" section — this lesson is that contract's explicit close: every boundary named there, now with the exact evidence from this capstone that motivates it.

An analogy: the finished model, with the list of "what the model doesn't simulate"

An architecture studio that hands over a perfectly finished scale model — with its five pieces assembled, every detail correct — also hands over, if honest, a clear list of what the model does not simulate: it has no working electrical wiring, it doesn't withstand a real earthquake, it has no plumbing that runs real water. None of those absences means the model is badly built — it means each of those pieces is a specialized trade, with its own engineer, hired separately when the building moves from model to real construction. This Iceberg lakehouse is that model: well assembled, with the five pieces fitting exactly as they should. This lesson is the list of specialized engineers Kiosko is going to need to hire next, each named precisely, none improvised.

Boundary 1: streaming and CDC as the change's real source → streaming-with-kafka-and-flink-guide

What it solves. P002's change — snacks/0.60 to health-snacks/0.68, effective 2026-08-15 — arrived, throughout this whole guide, as two fixed lists of dictionaries in Python: DIM_PRODUCT_V1 and DIM_PRODUCT_V2, hand-written, loaded with table.overwrite() or table.upsert() at the exact moment you chose to run the script. streaming-with-kafka-and-flink-guide teaches the opposite world: change data capture (CDC), where that same change arrives as a Kafka event at the exact instant it happened in Kiosko's source system — the inventory system that decided to recategorize the energy bar — with nobody writing a second Python list or running a script by hand.

This lakehouse's evidence. This module's lesson 4 captured snap_v1 immediately before applying V2 — a perfectly controlled sequence, because you decided when each step happened. In a system with real CDC, Iceberg would keep creating a new snapshot per write — that part doesn't change — but the write itself would arrive triggered by a Kafka consumer, processing events as the source system produces them, with no human running python3 kiosko_dim_product_time_travel.py at the right moment.

Boundary 2: published contracts and lineage → data-reliability-and-governance-guide

What it solves. This lakehouse has, thanks to Iceberg, something no previous guide in the ecosystem had with the same strength: an auditable history — every kiosko.dim_product snapshot is an immutable record of what changed and when, queryable with table.history(). But that history lives only inside the kiosko catalog, with no published contract telling a consuming team — BI, finance — which columns they can expect, with which types, with which quality guarantees. data-reliability-and-governance-guide teaches independently versioned data contracts, observability with automated alerts, and formal lineage between systems — mechanisms that operate on top of a catalog like this one, not inside it.

This lakehouse's evidence. kiosko.dim_product never had, in any lesson of this guide, an explicit validation that unit_cost is always positive, or that category belongs to a closed list of allowed values — Iceberg guarantees each column's type is correct (DoubleType, StringType), but it guarantees nothing about the business content. A published data contract, in data-reliability-and-governance-guide's sense, would make that guarantee explicit and independently verifiable, with no consumer needing to read this capstone's source code to trust the data.

Boundary 3: managed catalogs with real credentials → aws-core-services-guide

What it solves. This whole guide's eight modules ran on a SqlCatalog backed by SQLite, on the local filesystem — perfect for learning, perfect for $0 cost, and exactly what module 7 named, without implementing, as the real distance to production. aws-core-services-guide teaches the catalogs that really operate in production with real credentials: AWS Glue Data Catalog, S3 Tables — with IAM permissions, multi-tenant access control, and a single pointer to the current metadata really shared among different engines (Spark, Trino, Athena), not just declared as possible.

This lakehouse's evidence. kiosko_catalog.db is a single SQLite file on your disk — if two processes on two different machines needed to write to kiosko.dim_product at the same time, no mechanism in this lakehouse solves that; module 7 already named this as the exact guarantee a REST or Glue catalog does offer, with real concurrency control among distributed writers.

Boundary 4: scheduled orchestration → airflow-and-declarative-orchestration-guide

What it solves. Every operation in this capstone — every append(), every overwrite(), every upsert() — you triggered, running a script from your terminal, in the order you chose. This module's lesson 6's MERGE INTO/upsert() didn't run as a scheduled task in any pipeline — it ran because you executed python3 kiosko_native_merge_check.py at this specific moment. airflow-and-declarative-orchestration-guide takes exactly this same code, without changing a single line of the Iceberg logic, and turns it into a DAG task: managed retries if a write fails halfway through, sensors that wait for the source file to really exist before triggering the load, and alerts if something's still failing after retries.

This lakehouse's evidence. Nothing in this module's lesson 6 code stops two people, with no coordination, from running kiosko_native_merge_check.py twice the same day — each run would create its own kiosko.dim_product_merge_check table from scratch, or fail with TableAlreadyExistsError if the table already exists. An Airflow DAG would solve that ambiguity with a single source of truth about when and how many times each task runs — the same operational discipline no guide in this ecosystem built yet on its own, no exception.

Boundary 5: dbt versioning this same transformation as code → dbt-analytics-engineering-guide

What it solves. dim_product's change from V1 to V2 is written, in this capstone, as imperative Python inside a script — not as a declarative, versioned model, with automated tests and generated documentation. dbt-analytics-engineering-guide already taught how to automate exactly this same problem with dbt snapshot; what that guide couldn't teach yet — because Iceberg was, at that point in the ecosystem, this guide's territory — is the dbt-iceberg adapter, which lets a dbt project write directly onto Iceberg tables, with ref(), generic tests, and dbt docs generate operating on this capstone's same kiosko.dim_product, instead of a regular DuckDB table.

This lakehouse's evidence. This module's lesson 4 script repeats, every time it runs, the same business logic — load V1, capture snap_v1, apply V2 — with no declarative test automatically confirming P002's unit_cost changed exactly from 0.60 to 0.68 and not to another value by a typo. A dbt model over dbt-iceberg, with a generic accepted_values test or an equivalent singular test, would turn that manual verification — today solved with hand-written asserts in each closing project — into part of the project's own declarative infrastructure, versioned alongside the model.

Boundary 6: SQL performance in depth → advanced-sql-querying-guide

What it solves. Every query in this capstone — lesson 4's in-memory JOIN, lesson 5's store_id == 'S01' filter — ran with no lesson in this guide measuring, with EXPLAIN or an equivalent tool, the real execution plan behind those operations. advanced-sql-querying-guide teaches that topic in depth: execution plans, index tuning, the set theory behind every JOIN type — tools that become indispensable the moment a real lakehouse's volume exceeds what a learning demo needs to measure.

This lakehouse's evidence. This module's lesson 5 did measure something related — files_touched_s01 against files_touched_all, hidden partitioning's real file pruning — but that measurement is Iceberg-specific (which files a scan touches), not a general SQL execution-plan analysis. On kiosko.fact_orders_at_scale, with ten million rows, any query more complex than a simple store_id filter — a JOIN with aggregation, for example — would benefit exactly from the kind of analysis advanced-sql-querying-guide teaches, and that this guide, by design, never went deeper into.

Boundary 7: data FinOps in depth → cost-optimization-caching-guide

What it solves. This guide's module 7 already measured, with real evidence, that five nights of a redundant pipeline multiplied kiosko.dim_product's snapshot count by more than four — an operational hygiene problem, solved with expire_snapshots(). What that module deliberately didn't do was translate that growth into a dollar cost: how much it costs, on a real S3 bucket, to keep those redundant snapshots for a month, or how often to run maintenance given a specific budget. cost-optimization-caching-guide teaches that complete angle: data FinOps, not just technical hygiene.

This lakehouse's evidence. kiosko.fact_orders_at_scale, with ten million rows and two partition schemes coexisting, is exactly the kind of table where a poorly calibrated maintenance decision — expiring snapshots too often, or never expiring them — has a real, measurable storage cost. This guide measured that table in rows and in files touched; it never measured it in dollars, because that specific calculation — per-GB storage rates, per-request read costs — belongs, by design, to cost-optimization-caching-guide.

The complete map

flowchart TD
    K["Kiosko's lakehouse\n5 Iceberg tables, 1 catalog\nrun by hand, on your laptop"]

    K -->|"P002 still arrives as\na fixed value, not real CDC"| A["streaming-with-kafka-\nand-flink-guide"]
    K -->|"auditable history with no\npublished contracts or lineage"| B["data-reliability-and-\ngovernance-guide"]
    K -->|"local catalog, no\nreal credentials"| C["aws-core-services-guide"]
    K -->|"every operation triggered\nby hand, not scheduled"| D["airflow-and-declarative-\norchestration-guide"]
    K -->|"dbt-iceberg would version\nthis same transformation"| E["dbt-analytics-\nengineering-guide"]
    K -->|"no EXPLAIN or execution\nplan tuning"| F["advanced-sql-\nquerying-guide"]
    K -->|"snapshots measured in\nrows, never in dollars"| G["cost-optimization-\ncaching-guide"]

Why none of these seven boundaries invalidates what you already built

It's worth closing this lesson with the same discipline this guide's module 7 already used: naming a limitation isn't pointing out a mistake. Every pattern you built in this guide — real Iceberg tables with verified ACID, automatic snapshots on every commit, time travel with no history column at all, schema and partition evolution without rewriting a single file, two native merge paths converging on the same result — is the same pattern a production lakehouse uses at any scale, on any catalog, orchestrated by any system. None of this lesson's seven guides replaces what you learned — each one assumes you already know it, and builds on that foundation exactly like this module built on the seven before it.

Common mistakes

Thinking these seven boundaries mean you have to learn all of them before using Iceberg in a real job. What happens: someone finishes this lesson feeling Kiosko's lakehouse is "incomplete" until they master the seven sibling guides, and postpones any real use of Iceberg until then. Why it happens: seeing seven boundaries named at once feels overwhelming, as if they were seven prerequisites instead of seven possible paths. How to spot it: ask yourself whether a junior data engineer, in their first real job, needs to know CDC with Kafka or data FinOps before writing their first production Iceberg table — the answer, with the market evidence this guide's DESIGN doc cited from the start (a company asking literally for "Apache Iceberg or Delta Lake table formats on AWS S3"), is no: Iceberg with a managed catalog is, on its own, a hireable skill. How to fix it: these seven guides are paths you walk when the specific problem shows up — when you really need CDC, when you really need orchestration — not a list of prerequisites you have to exhaust before using what you already learned.

Confusing "this guide doesn't cover it" with "this guide got it wrong." What happens: someone notices the kiosko/SQLite catalog "isn't as robust as Glue," or that P002's change "isn't as realistic as CDC," and concludes this guide's design has a flaw. Why it happens: every boundary named in this lesson sounds, on first read, like a gap — it's easy to read "this doesn't solve it" as "this was solved wrong." How to spot it: check whether the pattern itself — not the infrastructure — is correct: table.overwrite() and table.upsert() apply P002's change in an ACID way, with the exact evidence you verified in lessons 4 and 6; a Glue catalog solving the same coordination need with different infrastructure doesn't make Iceberg's mechanism wrong, only that it operates at a different scale and with different governance. How to fix it: always separate "is Iceberg's pattern correct?" from "does this specific infrastructure scale to the production context I need?" — this lesson's seven boundaries are, mostly, the second question, not the first.

Looking for the seven sibling guides in the order they appear in this lesson, as if it were a mandatory syllabus. What happens: someone decides the correct next step is reading streaming-with-kafka-and-flink-guide first, then data-reliability-and-governance-guide, in this lesson's exact order, as if a single correct sequence existed. Why it happens: any numbered list gets read, instinctively, as a sequence — it's easy to forget this lesson's order is just the order the guide's original DESIGN doc named them in, not a mandatory pedagogical progression. How to spot it: ask yourself what specific problem you have today, in your own work or project — if your local catalog no longer suffices because you need to share it across several engines, aws-core-services-guide is your next step; if you need to schedule automated runs, it's airflow-and-declarative-orchestration-guide. How to fix it: choose the sibling guide based on the real problem in front of you, not on the order this lesson mentioned them in — all seven are independent of each other, with no declared dependency among them.

Exercises

Exercise 1 — Match each boundary to the specific evidence from this lakehouse that motivates it. Without looking at this lesson's sections, for each of the seven sibling guides, write in one sentence which specific part of Kiosko's lakehouse (a lesson, a table, a command) serves as evidence for why that boundary exists.

See solution

streaming-with-kafka-and-flink-guide → DIM_PRODUCT_V1/DIM_PRODUCT_V2, two fixed lists in Python, instead of a real CDC event. data-reliability-and-governance-guide → kiosko.dim_product with no published business validation on unit_cost or category. aws-core-services-guide → kiosko_catalog.db, a single local SQLite file, with no concurrency control among distributed writers. airflow-and-declarative-orchestration-guide → every script in this module, triggered by hand from the terminal, with no retries or sensors. dbt-analytics-engineering-guide → the V1→V2 logic written as imperative Python, with no declarative dbt model over dbt-iceberg. advanced-sql-querying-guide → no EXPLAIN on lesson 4's in-memory JOIN. cost-optimization-caching-guide → module 7's redundant snapshots, measured in quantity, never in dollars.

Exercise 2 — Choose, for your own context, which of the seven boundaries you'd solve first. With no single correct answer, choose one of the seven sibling guides and write, in 2-3 sentences, why it would be the most urgent for a real project you know or can imagine.

See solution

There's no single answer — the correct answer depends on the real problem. A reasonable example: if the imagined lakehouse already runs on an S3 bucket shared by several teams, with no managed catalog coordinating access, aws-core-services-guide would be the clearest priority — the exact same problem this guide's module 7 already named, without implementing, with evidence from the local kiosko catalog. Another equally valid example: if the business needed to react to product catalog changes within minutes, not the moment someone runs a script, streaming-with-kafka-and-flink-guide would become urgent before any other boundary.

Exercise 3 — Explain why data-modeling-for-analytics-guide doesn't appear on this lesson's map, despite being the most-cited guide in this whole module. In 2-3 sentences, explain the difference between a guide that's a source (data-modeling-for-analytics-guide) and a guide that's a next step (this lesson's seven).

See solution

data-modeling-for-analytics-guide doesn't appear on the "what Kiosko still needs" map because it isn't a forward boundary — it's the already-consumed source for this whole guide's problem: dim_product historized by hand, with valid_from/valid_to/is_current, and P002's canonical change (10.8/9.36), both already reproduced and verified in this module's lessons 3 and 4. This lesson's seven guides, instead, solve problems this lakehouse still doesn't have solved — CDC, contracts, managed catalogs, orchestration, dbt over Iceberg, performance SQL, FinOps — each with its own concrete evidence of why it's needed, not a problem already finished with the table format.

Summary and next step

This lesson closed the complete data-engineering-ecosystem map from this lakehouse's perspective: seven boundaries named precisely — real CDC, contracts and lineage, managed catalogs, scheduled orchestration, dbt over Iceberg, performance SQL, data FinOps — each with the exact sibling guide that solves it and the concrete evidence from this very capstone that motivates why it's needed. None of the seven invalidates what you built across this guide's eight modules — every pattern you learned is the same one a production lakehouse uses at any scale.

Before moving on you should be able to: name the seven sibling guides from memory, along with the specific problem each one solves; and explain why data-modeling-for-analytics-guide isn't part of this map, despite being the source for this whole module's problem.

Lesson 8 — this whole guide's final project — brings the eight complete modules together into a single script, start to finish, with the final verification that the total revenue (106.15) and P002's correct margin (10.8, via time travel) exactly match data-modeling-for-analytics-guide M8 and dbt-analytics-engineering-guide M8.

Resources

• streaming-with-kafka-and-flink-guide — CDC as a dimension change's real source, instead of a fixed value declared in Python. Sibling guide in this ecosystem.
• data-reliability-and-governance-guide — published data contracts and formal lineage, beyond the auditable history an Iceberg catalog already offers on its own. Sibling guide in this ecosystem.
• aws-core-services-guide — managed cloud catalogs (AWS Glue Data Catalog, S3 Tables), with real credentials and concurrency control, beyond the local kiosko/SQLite catalog. Sibling guide in this ecosystem.
• airflow-and-declarative-orchestration-guide — orchestrates every script in this lakehouse as a DAG task, with retries and sensors. Sibling guide in this ecosystem.
• dbt-analytics-engineering-guide — the dbt-iceberg adapter, to version this same transformation as a declarative dbt model over Iceberg tables. Sibling guide in this ecosystem.
• advanced-sql-querying-guide — execution plans (EXPLAIN) in depth, index tuning, beyond the partition file pruning this lakehouse already measured. Sibling guide in this ecosystem.
• cost-optimization-caching-guide — data FinOps: the real dollar cost of keeping snapshots, beyond the file count this guide's module 7 already measured. Sibling guide in this ecosystem.
• src/paths/data-engineering-ecosystem/VALIDACION.md — the market audit confirming this guide's verdict and the mandate to cover catalogs and compaction, the foundation for module 7 and this closing lesson. Internal repo document. In Spanish.