What Kiosko still needs

Description

Kiosko's warehouse works, end to end, and you proved it with evidence: bronze and silver rebuilt, a star with real history, two fact patterns no transactional model can solve, and an OBT published with the correct join already inside. It would be a mistake, however, to close this guide thinking that warehouse is "finished" in the sense a real data team would finish it. It isn't — and this lesson exists to be honest about exactly what it's missing, with no vagueness, naming every gap precisely along with the specific NIEVA Data Engineering ecosystem guide that solves it.

Connection to the module. This lesson adds not a single line of new code to Kiosko's warehouse — it's, deliberately, a map. Every row in the table that follows names a real, verifiable limitation of what you built in this guide's eight modules, and the exact sibling guide that solves it. It isn't a list of "interesting things you could learn later" — it's the explicit boundary this guide drew from its own design, module by module, that now, with the complete warehouse in front of you, finally makes sense to see together.

An analogy: someone who already knows how to design a house, not yet how to run the construction company

Someone who finishes a complete residential-architecture course and designs, with real judgment, a house's blueprints — foundation, layout, plumbing, electrical — already knows how to design a house. But that same person can't, yet, run a construction company building a hundred houses a year: they don't know how to coordinate twenty crews working in parallel, don't know what to do when the cement supplier falls behind mid-project, don't know how to version the blueprints when three different architects modify them the same week. None of those gaps invalidates what they learned — the house they know how to design is real, and the judgment they mastered (grain, shape, history, distribution of responsibility) is exactly the same judgment someone directing a hundred simultaneous projects at greater scale would need.

You finished this guide knowing how to design a complete dimensional warehouse: Kiosko's, with a declared grain, preserved history, correct point-in-time joins, and verified schema contracts. What follows — running a complete production warehouse's construction, with real volume, with a team, with orchestration and governance — is exactly what this ecosystem's sibling guides teach, each one deepening a specific thread that here, on purpose, was left superficial.

The map: the sibling guides, and what each one solves

# ecosystem_map.py
GAPS = [
    ("dbt-analytics-engineering-guide",
     "This warehouse's SQL lives in Python scripts run directly "
     "(con.execute(...)) -- nobody else on the team can version it, test it "
     "declaratively, or trace its lineage without reading the entire Python code."),
    ("airflow-and-declarative-orchestration-guide",
     "bronze -> silver -> gold runs by hand, in a single script, in the order "
     "YOU wrote it -- nothing schedules it to run alone every night, nor "
     "retries a failing step if dim_product_scd's MERGE cuts off halfway."),
    ("spark-and-distributed-processing-guide",
     "Kiosko's 40 orders and 32 events fit in memory on purpose -- a real "
     "retailer, with millions of orders and thousands of products changing "
     "category every quarter, would need to know when a cluster IS needed."),
    ("lakehouse-and-iceberg-guide",
     "dim_product_scd historizes by hand, with valid_from/valid_to columns and a "
     "MERGE INTO you wrote -- no native time travel, no ACID transactions "
     "across multiple writers, no formal catalog."),
    ("streaming-with-kafka-and-flink-guide",
     "P002's change arrived as a fixed snapshot (products_v2), loaded by hand "
     "-- in real production, that change would arrive via CDC from the "
     "catalog's transactional system, not as a Python file declared on purpose."),
    ("data-reliability-and-governance-guide",
     "validate_gold_schema() is a local function you run yourself -- there's no "
     "formal lineage, no data contracts published as an artifact, no column-level "
     "access control (who can see unit_cost, who only revenue)."),
    ("python-for-data-engineering-guide",
     "All this warehouse's code is plain Python, with no tests, no structured "
     "logging, no packaging -- the minimal glue code to fire off SQL, "
     "not DuckDB/Polars's full API as a daily work engine."),
    ("advanced-sql-querying-guide",
     "EXPLAIN got used only once (module 3) to COMPARE star vs snowflake -- "
     "never to tune a slow execution plan, nor to write a recursive CTE "
     "over a real production warehouse."),
]

print("=== Kiosko: what this warehouse does NOT solve, and who does ===\n")
for guide, gap in GAPS:
    print(f"{guide}")
    print(f"  -> {gap}\n")

What to expect. Running python3 ecosystem_map.py, the output is exactly this:

=== Kiosko: what this warehouse does NOT solve, and who does ===

dbt-analytics-engineering-guide
  -> This warehouse's SQL lives in Python scripts run directly (con.execute(...)) -- nobody else on the team can version it, test it declaratively, or trace its lineage without reading the entire Python code.

airflow-and-declarative-orchestration-guide
  -> bronze -> silver -> gold runs by hand, in a single script, in the order YOU wrote it -- nothing schedules it to run alone every night, nor retries a failing step if dim_product_scd's MERGE cuts off halfway.

spark-and-distributed-processing-guide
  -> Kiosko's 40 orders and 32 events fit in memory on purpose -- a real retailer, with millions of orders and thousands of products changing category every quarter, would need to know when a cluster IS needed.

lakehouse-and-iceberg-guide
  -> dim_product_scd historizes by hand, with valid_from/valid_to columns and a MERGE INTO you wrote -- no native time travel, no ACID transactions across multiple writers, no formal catalog.

streaming-with-kafka-and-flink-guide
  -> P002's change arrived as a fixed snapshot (products_v2), loaded by hand -- in real production, that change would arrive via CDC from the catalog's transactional system, not as a Python file declared on purpose.

data-reliability-and-governance-guide
  -> validate_gold_schema() is a local function you run yourself -- there's no formal lineage, no data contracts published as an artifact, no column-level access control (who can see unit_cost, who only revenue).

python-for-data-engineering-guide
  -> All this warehouse's code is plain Python, with no tests, no structured logging, no packaging -- the minimal glue code to fire off SQL, not DuckDB/Polars's full API as a daily work engine.

advanced-sql-querying-guide
  -> EXPLAIN got used only once (module 3) to COMPARE star vs snowflake -- never to tune a slow execution plan, nor to write a recursive CTE over a real production warehouse.

Eight lines, eight real gaps, eight guides. None of these limitations is a mistake in the warehouse you built — they are, exactly, the boundary this guide drew since module 1: "here you reach the correct dimensional model, genuinely executed; the production infrastructure surrounding it lives in the corresponding sibling guide."

Going deeper: one by one, the sibling guides

dbt-analytics-engineering-guide. This warehouse's SQL — MERGE INTO dim_product_scd, the point-in-time JOIN, mart_daily_sales_obt's aggregation — lives inside Python scripts, run with con.execute(open("file.sql").read()) or directly as inline text. Nobody else on a real team can version those business rules declaratively in Git, write a test confirming "category should never be null in dim_product_scd" without writing it by hand as an assert, or answer "which exact source table does the OBT's margin column come from?" without reading the Python code line by line. This sibling guide takes the same dimensional model you built here and turns it into a real dbt project: versioned SQL models, declarative tests (not_null, unique, relationships), and an automatic lineage graph.

airflow-and-declarative-orchestration-guide. This module's complete warehouse runs with a single command, python3 kiosko_analytics_warehouse.py, start to finish, in the exact order you wrote in the script. No system schedules it to run alone every early morning, no system automatically retries the second MERGE INTO if it fails from a transient memory issue, and no system detects bronze arrived incomplete before silver tries to process it. This sibling guide takes exactly this bronze→silver→gold sequence and turns it into a real DAG: sensors, retries managed by the orchestrator, and observability for every run.

spark-and-distributed-processing-guide. Kiosko's complete dataset — forty orders, thirty-two events — fits comfortably in any laptop's memory, and this guide declared that so, on purpose, since module 1: the question "does this need a cluster?" gets answered with a justified NO, not with a guess. But a real retailer, with millions of daily orders and a catalog of thousands of products constantly changing category, does cross the threshold where a MERGE INTO over a single DuckDB connection stops being enough. This sibling guide teaches when you DO need to distribute compute across several machines, and how to reason about a distributed execution plan for an SCD MERGE at scale.

lakehouse-and-iceberg-guide. dim_product_scd, as you built it in this module's lesson 4, historizes by hand: you decided the valid_from/valid_to/is_current columns, you wrote the MERGE INTO with its change-detection logic, and that table lives as a Parquet file or a simple DuckDB table, with no formal catalog tracking its versions. This sibling guide builds the layer that replaces that manual work with the engine's native time travel: Apache Iceberg or Delta Lake can answer "what did this table look like on August 14th?" with a simple version query, with nobody having to maintain valid_from/valid_to columns by hand — the same problem you solved in module 4, solved a different way, at the table-format level.

streaming-with-kafka-and-flink-guide. P002's change — from snacks/0.60 to health-snacks/0.68 — arrived at this warehouse as PRODUCTS_V2, a fixed snapshot declared directly in Python, loaded at a moment you decided. In real production, that change would arrive as an event from a transactional catalog system — someone, in some ERP, updates P002's record — propagated by Change Data Capture (CDC) toward the warehouse, with nobody having to manually prepare a second snapshot. This sibling guide teaches Kafka as a messaging system, Flink for continuous processing, and CDC as the typical production source for any SCD — this guide's module 4 already named this explicitly when building dim_product_scd, without implementing it.

data-reliability-and-governance-guide. validate_gold_schema(), built in module 7 and run again in this capstone, is a local contract: it lives inside the warehouse's own script, you run it yourself, and it knows nothing about what happens outside this specific pipeline. There's no formal lineage automatically tracing where each column comes from across a complete ecosystem of pipelines, no data contracts as a versioned artifact a consuming team could validate before using your data, and no granular access control — who can see unit_cost, each product's real cost, and who only the aggregated revenue you already saw in the OBT? This sibling guide builds exactly those capabilities, at a whole organization's scale.

python-for-data-engineering-guide. All this warehouse's code — kiosko.py, each lesson's scripts — is deliberately plain: .py files you run with python3 file.py, with no package structure, no pytest, no logging beyond a print(). This guide, the ecosystem's number two, already existed before this one — this guide only leaned on it as a style reference, without assuming it as a prerequisite — and it deepens exactly what stayed here as minimal glue code: real packaging, automated testing, structured logging, and DuckDB and Polars's full API as daily work engines, not as the tool that only fires off a SELECT every so often.

advanced-sql-querying-guide (linked, from another ecosystem). EXPLAIN, in this guide, got used only once — in module 3, to compare the cost of a one-hop JOIN (star) against a two-hop one (snowflake) — never to diagnose and tune a slow query in a real production warehouse. Window functions beyond ROW_NUMBER(), recursive CTEs, and professionally reading an execution plan to optimize (not just compare) live in this linked guide, from the SQL ecosystem, not the Data Engineering one.

Common mistakes

Feeling that Kiosko's warehouse "isn't useful" because it's missing these eight things. What happens: someone finishes this lesson with the feeling that everything built in the eight modules was, deep down, an incomplete toy exercise. Why it happens: seeing a list of eight gaps, all together, feels overwhelming — it seems like "almost everything is missing." How to spot it: if your conclusion is "so I didn't learn real dimensional modeling," reread this lesson's analogy — the house you learned to design is genuine, even though you don't yet know how to run the whole construction company. How to fix it: every pattern you built — grain, star, SCD, point-in-time join, accumulating snapshot, cumulative design, junk dimension, schema contract — is the exact same pattern a production warehouse uses at any scale, exactly as this guide's design warned since module 1. What's missing isn't "the correct model" — it's the production infrastructure around that model, and that's precisely what each sibling guide teaches.

Trying to learn all eight sibling guides at once. What happens: someone, motivated by this lesson's complete map, tries to open all eight guides in parallel, without finishing any of them thoroughly. Why it happens: seeing eight gaps explicitly named creates the urgency to "solve them all now." How to spot it: if you have eight browser tabs open with different guides and haven't completed even the first module of any of them, that's a sign of scattering, not progress. How to fix it: choose one guide, the one solving the gap that matters most for your specific situation — do you need to version your model as code for your current job? start with dbt; are you interested in native time travel? start with Iceberg — and finish it before opening the next one. This lesson's map is for orienting yourself, not for consuming in one sitting.

Assuming this lesson's table order is the order you have to learn them in. What happens: someone interprets that, because dbt-analytics-engineering-guide appears first in the list, it's the one "you have to" study first, regardless of their situation. Why it happens: a numbered or ordered list implicitly suggests a mandatory sequence. How to spot it: if you chose a sibling guide only because "it was first on the list," not because it solves a gap that matters to you specifically, reconsider your reason. How to fix it: this lesson's order simply follows the order each topic got mentioned throughout this guide's eight modules — it implies no priority nor mandatory sequence. Choose based on your own context: someone already working with Kafka at their current job probably gets more value from dbt-analytics-engineering-guide than from redoing CDC from scratch.

Exercises

Exercise 1 — Map your own gap. Think of a limitation in Kiosko's warehouse you yourself noticed during this guide's eight modules, one not explicitly named in this lesson's table (it can be something small). Identify which of the sibling guides it would correspond to, and justify your choice in 2-3 sentences.

See solution

There's no single correct answer — the exercise asks for personal reflection — but a reasonable example: "Kiosko's warehouse never validated that valid_from was always before valid_to in dim_product_scd, nor that two versions of the same product never overlapped in time." That specific limitation could be seen as an extension of validate_gold_schema() (which only validates schema, not business rules over the data) or, thought of as a rule that should apply consistently to any historized dimension across any pipeline in the organization, would fall within data-reliability-and-governance-guide.

Exercise 2 — Distinguish sibling guide from linked guide. Without looking at the previous sections, classify each of these four guides as sibling (part of the Data Engineering ecosystem) or linked (from another ecosystem, referenced because this warehouse needs it indirectly):

• (a) lakehouse-and-iceberg-guide
• (b) advanced-sql-querying-guide
• (c) streaming-with-kafka-and-flink-guide
• (d) python-for-data-engineering-guide

See solution

• (a) Sibling. Deepens the lakehouse's table format with native time travel, a thread explicitly opened in this guide's modules 4 and 5.
• (b) Linked. Belongs to the SQL ecosystem, referenced because EXPLAIN in this guide was deliberately introductory.
• (c) Sibling. Continues the CDC thread module 4 named when historizing dim_product_scd with a fixed snapshot instead of a real event.
• (d) Sibling. It's the Data Engineering ecosystem's guide number two — earlier than this one — deepening the production Python that stayed here as minimal glue code.

Exercise 3 — Argue why this guide didn't try to teach all eight things at once. Using the boundary criterion this guide declared in its own design (module 1), explain in 2-3 sentences why it would have been a pedagogical mistake for this guide to try to cover, even superficially, the eight sibling-guide topics within its own eight modules.

See solution

If this guide had tried to touch dbt, Airflow, Spark, Iceberg, Kafka, data governance, and production Python within its own eight modules, each topic would have received, at most, a superficial mention, without the real, verified code every concept it did cover in depth received — grain, star, SCD, point-in-time join, accumulating snapshot, cumulative design. Separating each topic into its own sibling guide, complete and terminal in itself, is what lets each one receive the same executable treatment dimensional modeling received here, instead of a two-minute demonstration with no assert behind it.

Summary and next step

In this lesson you stepped back from the code and traced the complete ecosystem map: seven sibling guides — dbt, Airflow, Spark, Iceberg, Kafka/Flink, data governance, production Python — plus one linked guide from another ecosystem — advanced SQL — each solving a specific, named gap in Kiosko's warehouse. No limitation named here invalidates what you built — it's, exactly, the boundary this guide drew in its own design, now seen complete with the finished warehouse in front of you.

Before closing the guide you should be able to: name, from memory, at least four of the eight guides named in this lesson and what each one solves; explain the difference between a sibling guide and a linked guide; and choose, with your own judgment — not by list order — which guide you'd follow first based on your own situation.

Lesson 8, the final mini-project closing the whole guide, brings the complete warehouse together one last time — bronze, silver, historized star, funnel, activity, OBT — in a single flow verified end to end: data-modeling-for-analytics-guide's definitive delivery.

Resources

• dbt Labs — "What is dbt?" — the official introduction to the project dbt-analytics-engineering-guide deepens. docs.getdbt.com/docs/introduction. In English.
• Apache Airflow — official documentation, airflow-and-declarative-orchestration-guide's starting point. airflow.apache.org/docs. In English.
• Apache Iceberg — official documentation, lakehouse-and-iceberg-guide's starting point, and the reference for the native time travel replacing this guide's manual SCD-2. iceberg.apache.org/docs/latest. In English.
• Databricks — "What is the medallion lakehouse architecture?" — the framework organizing this guide's complete warehouse, and the natural starting point toward lakehouse-and-iceberg-guide and data-reliability-and-governance-guide. docs.databricks.com/aws/en/lakehouse/medallion. In English.