What Kiosko still needs

Description

The system that exists at lesson 6's close is real: seven checks running in a single call, quarantine and alert triggered by the policy the contract itself declares, lineage mapped, role-based access applied, PII deterministically masked, and a catalog documenting Kiosko's four tables. And, at the same time, it's still a lab system in a very concrete sense: you triggered every run yourself, by hand, from your own terminal; the data it validates lives in a DuckDB table with no immutable history of its own; S04's broken file arrived as a CSV once, not as a continuous stream of events; and ACCESS_POLICY is a Python dictionary, not a real access policy enforced by any database or identity provider. This lesson isn't a critique of what you built — it's the honest map of the six exact boundaries this guide, from its own design, decided not to cross, and the precise name of the sibling guide that does cross each one, each with concrete evidence from this same system as its starting point.

Connection to the module. This guide's DESIGN named these boundaries from its first line, in the "NOT covered by" section. This lesson is that contract's explicit close: every boundary named there, now with this capstone's exact evidence motivating it.

An analogy: the finished alarm system, with the "what the system doesn't do" list

A company that installs a complete, functional fire alarm system — detectors, central panel, evacuation protocol — also delivers, if it's honest, a clear list of what that system does not do: it doesn't put out the fire on its own, it doesn't rebuild the building after a fire, it doesn't replace firefighters. None of those absences means the alarm system is poorly installed — it means each of those functions is a different discipline, with its own specialized team, hired separately when the need arises. Kiosko's trust system is that alarm system: well assembled, with every piece working exactly as it should. This lesson is the list of specialists Kiosko is going to need to hire next, each named precisely, none improvised.

Boundary 1: scheduled orchestration → airflow-and-declarative-orchestration-guide

What it solves. Every run_full_gate() run in this module, you triggered yourself, executing python3 run_s04_incident.py or python3 run_clean_day.py from your terminal, at the moment you chose. Nothing in this system runs automatically when a new file arrives in a shared Kiosko folder — nothing retries if the file doesn't exist yet, nothing schedules the daily run at a fixed hour, nothing notifies if the script simply never ran because the responsible person forgot. airflow-and-declarative-orchestration-guide takes this exact same code, with not a single line changed in run_full_gate(), and turns it into a DAG task: a sensor that waits for orders_YYYY-MM-DD.csv to really exist before triggering validation, managed retries if the connection to kiosko.duckdb fails halfway through, and an on_failure_callback that sends raise_alert()'s alert to a real channel when the gate reports any FAIL.

This system's evidence. Nothing in run_s04_incident.py stops two people, with no coordination, from running the script twice the same day on the same file — each run would generate its own report, with no single source of truth about whether S04 was already validated today or not. An Airflow DAG would resolve that ambiguity with a single scheduled task, executed once, with its own queryable run history — the same operational discipline no lesson in this guide built on its own.

Boundary 2: immutable table-format-level history → lakehouse-and-iceberg-guide

What it solves. orders_s04 lives, throughout this entire guide, as an ordinary DuckDB table: every CREATE OR REPLACE TABLE overwrites the earlier version with no trace of what was there before. If someone needed to answer, six months from now, "what did orders_s04 say exactly the day we ran the gate against the incident?", the answer would be "we don't know — the table has already been overwritten several times since then." lakehouse-and-iceberg-guide teaches the table format that solves exactly this: every write to an Iceberg table creates a new, immutable snapshot, queryable with time travel — the table that already has auditable history at the format level, over which a contract and quality system like this guide's would decide what it's allowed to write.

This system's evidence. Lesson 6's catalog documents orders_s04 with its associated contract, but has no way to answer "when did this table's schema last change?" or "which specific snapshot corresponds to the run that produced 6 failures?" — those questions, over an Iceberg table, would have an exact, verifiable answer (table.history()), with nobody having to keep that log by hand.

Boundary 3: real-time CDC as the source of change → streaming-with-kafka-and-flink-guide

What it solves. This entire guide's incident arrived as a CSV file, orders_2026-08-14.csv, delivered once, with all twelve rows already fixed from the moment S04 sent it. run_full_gate() runs on that complete file, after it already arrived — never on an individual event, at the exact instant it occurs. streaming-with-kafka-and-flink-guide teaches the opposite world: real change data capture (CDC) and streaming, where every S04 order would arrive as a Kafka event the moment it gets generated, and this module's same seven checks could apply per event, catching an invalid unit_price in seconds, not days after the whole file already accumulated and arrived late.

This system's evidence. Lesson 4's freshness failure — 47.58 hours of delay over a 24-hour SLA — is, in itself, a symptom that S04's data travels in batches, with all the delay that implies. A pipeline with real CDC would have, structurally, no equivalent notion of "file freshness": every event would arrive with its own origin timestamp, validated almost immediately — the same kind of broken file that arrived here as a CSV would arrive, in that world, as an individual event flagged invalid in real time.

Boundary 4: infrastructure observability, not just data observability → monitoring-observability-guide

What it solves. run_full_gate()'s seven checks observe data: null rows, duplicates, out-of-range prices, late files. None of them observes the infrastructure running those checks: how long run_full_gate() took to execute, whether the Python process used more memory than expected, whether kiosko.duckdb responded with normal latency, or whether the script crashed from an uncaught exception before even printing a result. monitoring-observability-guide teaches that entire second discipline: infrastructure metrics, distributed traces, system health dashboards — the observability principles are the same ones you already saw in this guide (something breaks, you need to find out, you need to be able to investigate why), but applied to the machine running the pipeline, not to the rows the pipeline processes.

This system's evidence. raise_alert(), in this guide, structures an alert about data — failure_count, severity, a sample of rows —; never about the process's state that generated it. If run_s04_incident.py hung halfway through the run from lack of memory, no mechanism in this guide would detect it — that's, precisely, the gap infrastructure observability closes, complementary to the data observability this guide taught end to end.

Boundary 5: column governance doesn't replace real IAM → aws-core-services-guide / cloud-security-and-guardrails-guide

What it solves. ACCESS_POLICY, in this guide, is a Python dictionary: {"analyst": [...], "finance": [...], "support": [...]}. apply_access_policy() applies it by filtering columns of a DataFrame already loaded in memory — there's no real user account, no password, no session token confirming whoever calls the function really is an analyst and not just someone who wrote "analyst" as an argument. aws-core-services-guide and cloud-security-and-guardrails-guide teach the layer missing before this: real IAM, with AWS accounts and roles, row- and column-level access policies enforced directly by the database engine (not by an if in Python), KMS keys for encryption at rest, and CloudTrail to audit who really accessed what, with a verifiable identity.

This system's evidence. build_role_view(customers_df, "analyst") trusts, with no verification at all, that the "analyst" string it receives is correct — anyone with access to the code could call build_role_view(customers_df, "finance") and see the phone numbers in plain text, with no credential involved. This lesson already warned about it since module 7: ACCESS_POLICY answers "given someone already authenticated as a known role, which columns should they see?" — authentication itself, the question of who that person is, is exactly these two sibling guides' territory.

Boundary 6: SQL performance and data FinOps → advanced-sql-querying-guide / cost-optimization-caching-guide

What it solves. No query in this guide — validate_referential_integrity()'s anti-join, reference_prices's group_by().agg() — ever got measured with EXPLAIN or with any execution plan analysis tool; at Kiosko's toy scale (twelve rows, forty rows, eight rows), no query needed optimizing. advanced-sql-querying-guide teaches that discipline in depth: execution plans, indexes, the theory behind each JOIN type — essential the moment a real data quality system's volume exceeds what a learning demo needs to measure. And no lesson in this guide translated the cost of maintaining contracts/orders_contract.yaml, running run_full_gate() every day, or storing quarantined rows indefinitely, into a dollar figure. cost-optimization-caching-guide teaches that entire angle: data FinOps, the real cost of sustaining a trust system at scale, not just its technical hygiene.

This system's evidence. run_full_gate(), run on twelve rows or on eight, takes a fraction of a second — no lesson in this guide had any real reason to measure its performance. At a real production volume, with thousands of daily files from dozens of stores, both every query's performance (advanced-sql-querying-guide) and the accumulated cost of running the complete system every day (cost-optimization-caching-guide) would become real business questions, not an implementation detail.

The complete map

flowchart TD
    K["Kiosko's trust system\nrun_full_gate() + quarantine + governance\nrun by hand, on your laptop"]

    K -->|"every run triggered\nby hand, with no DAG"| A["airflow-and-declarative-\norchestration-guide"]
    K -->|"orders_s04 with no real\nimmutable history"| B["lakehouse-and-\niceberg-guide"]
    K -->|"S04 arrived as a batch CSV,\nnot as a CDC event"| C["streaming-with-kafka-\nand-flink-guide"]
    K -->|"observes data, not the\ninfrastructure processing it"| D["monitoring-\nobservability-guide"]
    K -->|"ACCESS_POLICY is a dict,\nwith no real authentication"| E["aws-core-services-guide /\ncloud-security-and-guardrails-guide"]
    K -->|"no EXPLAIN, no cost\nmeasured in dollars"| F["advanced-sql-querying-guide /\ncost-optimization-caching-guide"]

Why none of these boundaries invalidates what you already built

It's worth closing this lesson with the same discipline module 7 already used: naming a limitation isn't pointing out a mistake. Every pattern you built in this guide — declarative tests generated from a versioned contract, real referential integrity against another table, anomaly detection with a really-calculated baseline, freshness and volume measured against a fixed clock, lineage mapped, quarantine instead of total rejection or silence, column governance by role with deterministic masking — is the same pattern a production data quality system uses at any scale, over any infrastructure, orchestrated by any system. None of this lesson's six boundaries replaces what you learned — each one assumes you already know it, and builds on that foundation exactly the way this module built on the seven earlier ones.

Common mistakes

Thinking these six boundaries mean you have to learn all of them before using what this guide taught in a real job. What happens: someone finishes this lesson feeling Kiosko's system is "incomplete" until they master the sibling guides, and postpones any real use of data contracts or declarative tests until then. Why it happens: seeing six boundaries named all at once feels overwhelming, as if they were six prerequisites instead of six possible paths. How to spot it: ask yourself whether a junior data engineer, on their first real job, needs to master Airflow or AWS IAM before they can write their first productive pa.DataFrameModel — the answer, with the market evidence this guide's DESIGN cited from the start (Data Quality 43%, Monitoring 31%, Governance 24% according to InterviewStack), is no: declarative tests and data contracts are, on their own, a hireable skill. How to fix it: these six guides are paths you walk when the specific problem shows up — when you really need orchestration, when you really need real IAM —, not a list of prerequisites you have to exhaust before using what you already learned.

Confusing "this guide doesn't cover it" with "this guide got it wrong." What happens: someone notices ACCESS_POLICY "isn't as robust as real IAM," or that running the gate by hand "isn't as realistic as a DAG," and concludes this guide's design has a flaw. Why it happens: every boundary named in this lesson sounds, on first reading, like a shortfall — it's easy to read "this doesn't solve it" as "this got solved badly." How to spot it: check whether the pattern itself — not the infrastructure — is correct: mask_pii() really produces deterministic hashes, verified in module 7; check_price_baseline() catches the dollars-to-cents bug with real evidence. That a real IAM system solves the same authentication need with different infrastructure doesn't make ACCESS_POLICY's mechanism wrong — only that it operates at a different scale and with a different guarantee. How to fix it: always separate "is the pattern correct?" from "does this specific infrastructure scale to the production context I need?" — this lesson's six boundaries are, mostly, the second question, not the first.

Exercises

Exercise 1 — Match each boundary with the specific evidence from this system motivating it. Without looking at this lesson's sections, for each of the six sibling guides (or guide pairs), write in one sentence what specific part of Kiosko's system — a lesson, a function, a result — serves as evidence for why that boundary exists.

See solution

airflow-and-declarative-orchestration-guide → every script in this module, triggered by hand from the terminal, with no DAG or retries at all. lakehouse-and-iceberg-guide → orders_s04 as an ordinary DuckDB table, with no snapshots or time travel. streaming-with-kafka-and-flink-guide → orders_2026-08-14.csv, a batch file delivered once, instead of individual CDC events. monitoring-observability-guide → raise_alert() structuring alerts about data, never about the process's state that generated them. aws-core-services-guide/cloud-security-and-guardrails-guide → build_role_view("analyst") trusting, with no verification at all, a role string with no real authentication. advanced-sql-querying-guide/cost-optimization-caching-guide → no query in this guide measured with EXPLAIN, no cost of maintaining the system translated into dollars.

Exercise 2 — Choose, for your own context, which of the six boundaries you'd solve first. With no single correct answer, choose one of the sibling guides and write, in 2-3 sentences, why it would be the most urgent for a real project you know or can imagine.

See solution

There's no single answer — it depends on the real problem. A reasonable example: if the imagined system already runs in production but every validation depends on someone remembering to manually run a script every morning, airflow-and-declarative-orchestration-guide would be the clearest priority — the exact same problem this lesson already named, with evidence that no run in this module triggered itself. Another equally valid example: if the company's security team already requires auditable database-level access control, aws-core-services-guide/cloud-security-and-guardrails-guide would become urgent before any other boundary, because ACCESS_POLICY as a Python dictionary wouldn't pass any real security audit.

Exercise 3 — Explain why dbt-analytics-engineering-guide and data-modeling-for-analytics-guide don't appear in this lesson's map, despite being mentioned in several earlier modules of this guide. In 2-3 sentences, explain the difference between a guide that's a source (those two) and a guide that's a forward boundary (this lesson's six).

See solution

data-modeling-for-analytics-guide and dbt-analytics-engineering-guide don't appear in the "what Kiosko still needs" map because they aren't forward boundaries — they're sources already consumed: data-modeling-for-analytics-guide built kiosko.duckdb and dim_store/dim_product, which this guide read with no redesign of the warehouse; dbt-analytics-engineering-guide already taught data_tests: inside a dbt project, the layer this guide explicitly named as a boundary in module 2, not here — this guide deliberately built a reusable test layer outside dbt. This lesson's six guides, by contrast, solve problems this system still doesn't have solved — orchestration, immutable history, CDC, infrastructure observability, real IAM, performance and cost —, each with its own concrete evidence of why it's needed, not a problem already finished being solved in an earlier module of this same guide.

Summary and next step

This lesson closed the data-engineering-ecosystem's complete map from this trust system's perspective: six boundaries named precisely — orchestration, immutable history, real-time CDC, infrastructure observability, real IAM, performance and data FinOps —, each with the exact sibling guide that solves it and this same capstone's concrete evidence motivating why it's needed. None of the six invalidates what you built across this guide's eight modules — every pattern you learned is the same one a production data quality system uses at any scale.

Before moving on you should be able to: name the six boundaries from memory, along with the sibling guide (or guide pair) solving each one; and explain why data-modeling-for-analytics-guide and dbt-analytics-engineering-guide aren't part of this map, despite being cited in earlier modules of this guide.

Lesson 8 — this entire guide's final project — brings lessons 4 and 5's two complete runs, plus lesson 6's governance layer, together into a single script, with automatic assertions confirming, one last time, this entire guide's two central numbers: 6 failures on S04, 0 failures on the clean day.

Resources

• airflow-and-declarative-orchestration-guide — orchestrates every script in this system as a DAG task, with sensors, retries, and on_failure_callback. Sibling guide in this ecosystem.
• lakehouse-and-iceberg-guide — the table with format-level immutable history a contract system like this one would lean on in production. Sibling guide in this ecosystem.
• streaming-with-kafka-and-flink-guide — CDC as the real source of change, instead of a batch CSV file delivered once. Sibling guide in this ecosystem.
• monitoring-observability-guide — infrastructure observability, beyond the data observability this guide taught end to end. Sibling guide in this ecosystem.
• aws-core-services-guide — real IAM, with accounts and credentials, beyond ACCESS_POLICY as a Python dictionary. Sibling guide in this ecosystem.
• cloud-security-and-guardrails-guide — row- and column-level access policies enforced by the database engine, KMS, CloudTrail. Sibling guide in this ecosystem.
• advanced-sql-querying-guide — execution plans (EXPLAIN) in depth, index tuning, beyond this guide's toy queries. Sibling guide in this ecosystem.
• cost-optimization-caching-guide — data FinOps: the real dollar cost of sustaining a trust system at scale. Sibling guide in this ecosystem.
• src/paths/data-engineering-ecosystem/VALIDACION.md — the market audit confirming this guide's verdict and the mandate to cover contracts, governance, and row/column-level access. Internal repo document. In Spanish.
• This guide's DESIGN — the complete "NOT covered by" section, the source of this lesson's six boundaries. src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.