What Kiosko still needs

Description

The kiosko_analytics/ project that exists by the end of lesson 6 is real: twelve models, one snapshot, thirty-three data_tests, automatically generated documentation, all reproducible with a single dbt build. And, at the same time, it's still a toy project in one very concrete sense: it runs on your laptop, over a kiosko.duckdb file of a few megabytes, triggered by hand from your own terminal, over data that never changes on its own. This lesson isn't a criticism of what you built — it's the honest map of the seven exact boundaries this guide, by its own design, decided not to cross, and the precise name of the sibling guide that does cross each one, each with concrete evidence from this same project as its starting point.

Connection to the module. Module 7's lesson 7 already did this exercise once, at a smaller scale: it named airflow-and-declarative-orchestration-guide as the explicit bridge toward orchestration, with evidence from the Required var 'run_date' not found error. This lesson repeats that same exercise, now over the complete twelve-model project, and extends it to the six remaining boundaries this guide's design named from its very first line.

An analogy: the finished house, with the "what the builder doesn't include" list

When someone hands over a newly built house, they also hand over — if the builder is honest — a clear list of what the warranty doesn't cover: the garden's irrigation system, solar panel installation, perimeter security. None of those absences mean the house is badly built — it means each of those pieces is a specialized trade, with its own contractor, hired separately when needed. This dbt project is that house: well built, livable, with everything a versioned dimensional warehouse needs. This lesson is the list of specialized contractors Kiosko is going to need to hire later, each named precisely, none improvised.

Boundary 1: production orchestration → airflow-and-declarative-orchestration-guide

What it solves. Every dbt build in this whole guide, you triggered by hand, from your terminal, passing --vars with the correct date every time. Module 7's lesson 7 already demonstrated, with the real Required var 'run_date' not found error, what happens the day someone — or nothing, an unsupervised process — forgets that step. airflow-and-declarative-orchestration-guide takes this exact same dbt build, with no single line of code changed, and turns it into a DAG task: run_date calculated automatically from when the task runs, managed retries if something fails, sensors that wait for Kiosko's raw files to really exist before triggering the run, and alerts if something keeps failing after the retries.

This project's evidence. The twelve models you built in this module unintentionally multiplied the surface of what a human has to remember by hand: it's not just fact_orders that needs run_date anymore — now fact_store_activity and mart_daily_sales_obt also depend, indirectly, on that variable being passed correctly. The bigger a dbt project grows, the more expensive the "someone remembers" discipline becomes — the exact reason orchestration stops being optional.

Boundary 2: distributed computing → spark-and-distributed-processing-guide

What it solves. Kiosko's complete warehouse — twelve models, hundreds of rows in total — ran in under a second, inside a kiosko.duckdb file that fits on any laptop. DuckDB, this whole guide's engine, processes data in-process, in a single machine's memory — a design decision that works perfectly at this scale, and stops working as soon as volume exceeds what one machine can hold in memory. spark-and-distributed-processing-guide teaches when that limit really gets crossed, and how the same transformation — the same kind of JOIN, the same kind of aggregation you already wrote in mart_daily_sales_obt — gets distributed across several nodes of a cluster.

This project's evidence. fact_store_activity used a window function (array_agg(...) OVER (...)) over 21 rows — instant at this scale. That same window function, over billions of real event rows, demands a completely different distributed architecture, not just "the same query but slower" — the exact moment spark-and-distributed-processing-guide becomes necessary, not optional.

Boundary 3: native lakehouse time travel → lakehouse-and-iceberg-guide

What it solves. This guide's module 5 built dim_product_snapshot: SCD type 2, automated, with dbt_valid_from/dbt_valid_to/dbt_scd_id columns dbt manages for you. That historization works — you verified it, number for number, in every module since then — but it's a dbt convention over an ordinary DuckDB table, not a property of the storage format itself. lakehouse-and-iceberg-guide teaches Apache Iceberg and Delta Lake: table formats where time travel — querying "what did the data look like a week ago" — is a native storage capability, with ACID schema evolution and versioned catalogs, with no project needing to maintain its own metadata columns.

This project's evidence. SELECT * FROM dim_product_snapshot WHERE dbt_valid_from <= '2026-08-10' AND (dbt_valid_to IS NULL OR dbt_valid_to > '2026-08-10') — the same kind of point-in-time query data-modeling-for-analytics-guide already taught — works, but it requires you to know exactly which columns to query and how. Iceberg solves the same question ("what did my data look like at a specific moment?") as a function of the engine, not as a manual query over columns a specific project decided to maintain.

Boundary 4: streaming and CDC → streaming-with-kafka-and-flink-guide

What it solves. Every piece of Kiosko's data across this whole guide is a fixed file — orders_2026-08-03.csv, products_v2.csv, with none of them changing while you run your commands. dbt snapshot, dbt run, dbt build: all three get triggered by hand, over a state of the world that sits still until you decide to look at it again. streaming-with-kafka-and-flink-guide teaches the opposite world: data that arrives continuously, one event at a time, with change data capture (CDC) replacing the question "what changed since the last time I looked?" — the same question dbt's snapshot answers in batch, every time someone runs dbt snapshot — with a flow that never stops running.

This project's evidence. P002's change (snacks → health-snacks) module 5 historized arrived as a new file, products_v2.csv, that you added by hand at a moment you chose. In a system with real CDC, that same change would have arrived as a Kafka event at the exact instant it happened in Kiosko's source system, with nobody needing to create any new file or run any command.

Boundary 5: governance and reliability at scale → data-reliability-and-governance-guide

What it solves. dbt docs generate, in module 7, produced this project's complete lineage: which model depends on which, which columns are documented. That lineage is real and useful — and it's a single project's lineage, the one running on your machine. data-reliability-and-governance-guide teaches data contracts published as independently versioned artifacts, observability with automatic alerts when something drifts, and row- and column-level access control — mechanisms that operate between systems and teams, not inside a single repository.

This project's evidence. Kiosko's thirty-three data_tests are an implicit contract: if dbt test passes, the team trusts the data is correct. A published data contract, in data-reliability-and-governance-guide's sense, makes that contract explicit and independently versioned — so a consuming team, on another system, can trust a documented guarantee without having to read kiosko_analytics/'s source code to know it.

Boundary 6: deep performance SQL → advanced-sql-querying-guide

What it solves. Every JOIN in this project — mart_daily_sales_obt's four, fact_orders's two — ran in milliseconds, over data that fits comfortably in memory. EXPLAIN was already used, in depth, in data-modeling-for-analytics-guide — this guide deliberately doesn't repeat it. advanced-sql-querying-guide teaches the complete topic: execution plans in depth, index tuning, the set theory behind every JOIN type, over volumes where performance stops being free.

This project's evidence. fact_store_activity uses three separate window functions — two per array column, once for the 7-day window and once for the 30-day one — over a table of barely 21 rows. At that scale, the cost is invisible. Over millions of rows, those same windows — repeated, not shared across columns, as they were left written in this module's lesson 3 for pedagogical clarity — would carry a real cost worth measuring and optimizing, exactly the kind of analysis advanced-sql-querying-guide teaches.

Boundary 7: production Python → python-for-data-engineering-guide

What it solves. This whole guide — the eight modules, every line of code — is almost entirely declarative SQL, configuration YAML, and Jinja for the macros. The only "imperative code" you wrote was two Jinja macros (calculate_revenue, is_positive), never a single line of production Python. python-for-data-engineering-guide teaches exactly what this guide deliberately avoided: Python project packaging, tests with pytest, structured logging, and DuckDB/Polars as a daily-work API instead of an engine behind dbt.

This project's evidence. Every time you verified a result in this guide with a short script (import duckdb; con.sql(...)), you were using exactly the pattern python-for-data-engineering-guide goes deep on — but as a one-off verification tool, never as the transformation engine itself. That boundary — Python to verify, dbt to transform — is the same one this guide's design drew from its very first line, and it stays true through the last module.

The complete map

flowchart TD
    K["kiosko_analytics/\n12 models, 1 snapshot, 33 data_tests\nrun by hand, on your laptop"]

    K -->|"who triggers dbt build\nevery day, unsupervised"| A["airflow-and-declarative-\norchestration-guide"]
    K -->|"when in-process DuckDB\nno longer scales"| B["spark-and-distributed-\nprocessing-guide"]
    K -->|"native storage time travel,\nnot dbt columns"| C["lakehouse-and-\niceberg-guide"]
    K -->|"data arriving\ncontinuously, not in batch"| D["streaming-with-kafka-\nand-flink-guide"]
    K -->|"published contracts,\ncross-system observability"| E["data-reliability-and-\ngovernance-guide"]
    K -->|"execution plans,\nreal-scale tuning"| F["advanced-sql-\nquerying-guide"]
    K -->|"production Python,\npackaging, pytest"| G["python-for-data-\nengineering-guide"]

Why none of these seven boundaries invalidates what you already built

It's worth closing this lesson with the same discipline module 7 already used: naming a limitation isn't pointing out a mistake. Every pattern you built in this guide — sources and staging separated from marts, ref() resolving the DAG, generic and singular tests, a snapshot automating SCD type 2, an incremental model with proven idempotency, reusable macros, generated documentation and lineage — is the same pattern a production dbt project uses at any scale, over any volume, orchestrated by any system. None of this lesson's seven guides replaces what you learned — each one assumes you already know it, and builds on that foundation exactly as this module built on the seven before it.

Common mistakes

Thinking these seven boundaries mean you have to learn all of them before using dbt in a real job. What happens: someone finishes this lesson feeling like Kiosko's project is "incomplete" until they master all seven sibling guides, and postpones any real dbt use until then. Why it happens: seeing seven boundaries named all at once feels overwhelming, as if they were seven prerequisites instead of seven possible paths. How to spot it: ask yourself whether a junior analytics engineer, at their first real job, needs to know Iceberg or Kafka before writing their first production dbt model — the answer, with the market evidence this guide's design cited from the start, is no: dbt Core with a cloud warehouse is, by itself, a hireable skill. How to fix it: these seven guides are paths you travel when the specific problem shows up — when volume really exceeds what in-process DuckDB can hold, when orchestration is genuinely needed — not a list of prerequisites you have to exhaust before using what you already learned.

Confusing "this guide doesn't cover it" with "this guide did it wrong". What happens: someone notices fact_store_activity would in theory run slowly at large scale, or that dbt's snapshot "isn't as good as Iceberg," and concludes this guide's design has a flaw. Why it happens: every boundary named in this lesson sounds, on first read, like a shortcoming — it's easy to read "this doesn't solve it" as "this was solved wrong." How to spot it: check whether the pattern itself — not the scale — is correct: dim_product_snapshot correctly historizes every P002 change, with the exact evidence you verified module by module; Iceberg solving the same problem with a different architecture doesn't make dbt's snapshot wrong, it just solves the problem at a different scale and with a different philosophy. How to fix it: always separate "is the pattern correct?" from "does this specific tool scale to the volume I need?" — this lesson's seven boundaries are, mostly, the second question, not the first.

Looking for the seven sibling guides in the order they appear in this lesson, as if it were a mandatory syllabus. What happens: someone decides the correct next step is reading airflow-and-declarative-orchestration-guide first, then spark-and-distributed-processing-guide, in this exact same order, as if a single correct sequence existed. Why it happens: any numbered list instinctively reads as a sequence — it's easy to forget this lesson's order is only the order the guide's original design named them in, not a mandatory pedagogical progression. How to spot it: ask yourself what specific problem you have today, in your own work or project — if you need to schedule automated runs, airflow-and-declarative-orchestration-guide is your next step; if your data volume no longer fits on a laptop, it's spark-and-distributed-processing-guide. How to fix it: choose the sibling guide based on the real problem in front of you, not on the order this lesson mentioned them in — all seven are independent of each other, with no declared dependency between any of them.

Exercises

Exercise 1 — Match each boundary with the specific evidence from this project that motivates it. Without looking at this lesson's sections, for each of the seven sibling guides, write in one sentence what specific part of kiosko_analytics/ (a module, a mart, a command) serves as evidence for why that boundary exists.

See solution

airflow-and-declarative-orchestration-guide → the Required var 'run_date' not found error from modules 6 and 7. spark-and-distributed-processing-guide → fact_store_activity's window functions, instant over 21 rows. lakehouse-and-iceberg-guide → dim_product_snapshot's dbt_valid_from/dbt_valid_to columns, managed by dbt convention, not by the storage. streaming-with-kafka-and-flink-guide → products_v2.csv, a file you added by hand, instead of a real-time CDC event. data-reliability-and-governance-guide → the thirty-three data_tests, a single project's implicit contract, not published or independently versioned. advanced-sql-querying-guide → fact_store_activity's three repeated array_agg(...) OVER (...), with no EXPLAIN at all measuring their real cost. python-for-data-engineering-guide → the short verification scripts (import duckdb) used in every module, never as the transformation engine.

Exercise 2 — Choose, for your own context, which of the seven boundaries you'd solve first. With no single correct answer, choose one of the seven sibling guides and write, in 2-3 sentences, why it would be the most urgent for a real project you know or imagine.

See solution

There's no single answer — the correct one depends on the real problem. One reasonable example: if the imagined project already runs on a cloud warehouse shared by a five-person team, with no automated process updating it, airflow-and-declarative-orchestration-guide would be the clearest priority — the exact same problem this guide's module 7 lesson 7 already demonstrated with evidence. Another equally valid example: if the data grew to tens of millions of rows, spark-and-distributed-processing-guide would become urgent before any other boundary, because no orchestration pattern solves a volume problem in-process DuckDB can no longer hold.

Exercise 3 — Explain why data-modeling-for-analytics-guide doesn't show up on this lesson's map, despite being the most cited guide in this whole module. In 2-3 sentences, explain the difference between a guide that's a source (data-modeling-for-analytics-guide) and a guide that's a next step (this lesson's seven).

See solution

data-modeling-for-analytics-guide doesn't show up on the "what Kiosko still needs" map because it isn't a boundary looking forward — it's the already-consumed source of the whole dimensional model this module ported: every mart in this module (dim_category, dim_order_flags, fact_sessions, fact_store_activity, mart_daily_sales_obt) comes from there, already designed and verified, with no piece of that specific guide still pending. This lesson's seven guides, by contrast, solve problems this project doesn't have solved yet — orchestration, distribution, native time travel, streaming, governance, performance at scale, production Python — each with its own concrete evidence for why it's needed, not a model that already finished being ported.

Summary and next step

This lesson closed the complete data-engineering-ecosystem map from this dbt project's perspective: seven boundaries named precisely — orchestration, distributed computing, native time travel, streaming/CDC, governance and reliability, performance SQL, production Python — each with the exact sibling guide that solves it and the concrete evidence from this same project motivating why it's needed. None of the seven invalidates what you built across this guide's eight modules — every pattern you learned is the same one a production dbt project uses at any scale.

Before moving on you should be able to: name the seven sibling guides from memory, along with the specific problem each one solves; and explain why data-modeling-for-analytics-guide isn't part of this map, despite being the source of all this module's work.

Lesson 8 — the whole guide's final mini-project — brings all eight complete modules together into a single flow, start to finish, with Kiosko project's eighth and final version-control commit.

Resources

• airflow-and-declarative-orchestration-guide — orchestrates this guide's dbt build as a DAG task, with automatic run_date, retries, and sensors. This ecosystem's sibling guide.
• spark-and-distributed-processing-guide — when volume demands the transformation get distributed, beyond in-process DuckDB. This ecosystem's sibling guide.
• lakehouse-and-iceberg-guide — table formats with native engine time travel (Apache Iceberg, Delta Lake), the production replacement for dbt's snapshot. This ecosystem's sibling guide.
• streaming-with-kafka-and-flink-guide — CDC as the source of a real-time incremental pipeline, beyond dbt run/dbt snapshot's batches. This ecosystem's sibling guide.
• data-reliability-and-governance-guide — published data contracts and observability at scale, beyond a single project's lineage that dbt docs generate produces. This ecosystem's sibling guide.
• advanced-sql-querying-guide — execution plans (EXPLAIN) in depth, index tuning, set theory applied to performance SQL. This ecosystem's sibling guide.
• python-for-data-engineering-guide — production Python: packaging, tests with pytest, structured logging, DuckDB/Polars as a daily-work API. This ecosystem's sibling guide.
• dbt Labs — "What is analytics engineering?," already cited in module 1, the same definition backing why each of these seven boundaries is a distinct specialized trade, not a natural extension of dbt itself. getdbt.com/blog/what-is-analytics-engineering. In English.