Module 8: Project Kioskos Distributed Pipeline

What Kiosko still needs

Next →

Description

Kiosko's distributed pipeline works, end to end, and you proved it with evidence across the previous four lessons: 26,537,500.00 in revenue verified via two independent paths, every design decision justified with measured evidence, and an honest verdict that not even this guide's synthetic dataset justified Spark's cost on its own. It would be a mistake, though, to close this guide thinking that pipeline is "finished" in the sense a real data team would consider it complete. It isn't — and this lesson exists to be honest about exactly what it's missing, with no vagueness, naming every gap precisely along with the specific guide in NIEVA's Data Engineering ecosystem that resolves it.

Connection to the module. This lesson adds not one line of new code to the pipeline — it's, deliberately, a map. Every row in the table that follows names a real, verifiable limitation of what you built in modules 1 through 8, and the exact sibling guide resolving it. It isn't a list of "interesting things you could learn later" — it's the explicit boundary this guide's DESIGN doc drew from its own design, module by module, and that now, with the complete capstone in front of you, finally makes sense to see all together.

An analogy: knowing how to run the machinery, not yet how to run the whole factory

Someone who learns to operate, with genuine mastery, a specific piece of industrial machinery — knows its controls, knows how to read its gauges, knows when to stop it before something breaks — already has a real, valuable skill. But that same person doesn't yet know how to schedule the factory's complete shift: doesn't know how to coordinate preventive maintenance with production, doesn't know what to do when the raw-material supplier changes the delivery schedule, doesn't know how to certify the complete line meets the plant's safety standards. None of those gaps invalidates what they learned — the machine they know how to operate is real, and the discipline they mastered (reading gauges, understanding the mechanism, knowing when to step in) is exactly the same discipline someone running the entire factory would need, at a larger scale.

You finished this guide knowing how to operate Spark with genuine mastery: the DataFrame API, shuffle, joins with real criteria, Catalyst, partitioned Parquet, pandas_udf, and the honest criterion for when that machinery is needed. What comes next — scheduling the pipeline to run on its own, versioning the transformation as team code, giving time travel to the data Spark writes, processing continuous data instead of batches, governing access and quality at organization scale, and operating a real cluster with its real cost — is exactly what this ecosystem's sibling guides teach, each one going deep on a specific thread deliberately left out of scope here.

The map: the sibling guides, and what each one resolves

# ecosystem_map.py
GAPS = [
    ("airflow-and-declarative-orchestration-guide",
     "This capstone's complete pipeline got run by hand, from the terminal, in every "
     "lesson. No system schedules it to run on its own every night, no system waits "
     "for kiosko_orders_at_scale.csv to exist before triggering the run, and no system "
     "retries or alerts if a complete run didn't happen."),
    ("dbt-analytics-engineering-guide",
     "fact_orders_at_scale got built with PySpark code living in a script -- with no "
     "explicit versioning of its business rules as a declarative model, no 'dbt test' "
     "style tests an analyst with no Python experience could write, and no automatic "
     "lineage. The dbt-spark adapter exists exactly to connect dbt over this same "
     "engine, without building it here."),
    ("lakehouse-and-iceberg-guide",
     "fact_orders_at_scale_m8.parquet is flat Parquet, partitioned by folders: it has no "
     "time travel (you can't ask 'what did this dataset look like before the last "
     "rebuild'), doesn't support ACID transactions if two processes wrote to it at the "
     "same time, and there's no formal catalog beyond it being a folder on disk."),
    ("streaming-with-kafka-and-flink-guide",
     "This entire guide processed data in batches -- kiosko_orders_at_scale.csv already "
     "existed, complete, before the pipeline started. If Kiosko wanted a sales report "
     "updated in real time instead of a complete run, neither the DataFrame API in batch "
     "mode nor any code from this guide solves that by design."),
    ("data-reliability-and-governance-guide",
     "This capstone's pipeline has no formal lineage system beyond the comments in the "
     "code itself, no versioned data contract a consuming team could validate before "
     "using fact_orders_at_scale, and no granular access control -- who can see each "
     "product's real unit_cost, and who should only see aggregated revenue?"),
    ("aws-core-services-guide",
     "This entire pipeline ran on local[*], on a laptop, at $0 cost and with no cloud "
     "account at all. The day lesson 6's Scenario 3 (16.4 TB, IT DEPENDS) becomes real "
     "for Kiosko, EMR or Databricks are the managed Spark that replaces local[*] with a "
     "real cluster -- never built here."),
    ("cost-optimization-caching-guide",
     "This module's lesson 6 closed the IT DEPENDS zone with an explicit instruction: "
     "'measure the real cost of a cluster against a bigger node' -- but this entire "
     "guide never built that measurement. That is, specifically, this linked guide's "
     "FinOps discipline."),
    ("advanced-sql-querying-guide",
     "The .explain() you read in modules 5 and 6 is Catalyst's physical plan over "
     "distributed DataFrames -- a different topic from a traditional relational "
     "engine's (PostgreSQL, MySQL) EXPLAIN, with its own index tuning and execution "
     "plans."),
]

print("=== Kiosko's distributed pipeline: what it does NOT resolve, and who does ===\n")
for guide, gap in GAPS:
    print(f"{guide}")
    print(f"  -> {gap}\n")

What to expect. Running python3 ecosystem_map.py, the output is exactly this:

=== Kiosko's distributed pipeline: what it does NOT resolve, and who does ===

airflow-and-declarative-orchestration-guide
  -> This capstone's complete pipeline got run by hand, from the terminal, in every lesson. No system schedules it to run on its own every night, no system waits for kiosko_orders_at_scale.csv to exist before triggering the run, and no system retries or alerts if a complete run didn't happen.

dbt-analytics-engineering-guide
  -> fact_orders_at_scale got built with PySpark code living in a script -- with no explicit versioning of its business rules as a declarative model, no 'dbt test' style tests an analyst with no Python experience could write, and no automatic lineage. The dbt-spark adapter exists exactly to connect dbt over this same engine, without building it here.

lakehouse-and-iceberg-guide
  -> fact_orders_at_scale_m8.parquet is flat Parquet, partitioned by folders: it has no time travel (you can't ask 'what did this dataset look like before the last rebuild'), doesn't support ACID transactions if two processes wrote to it at the same time, and there's no formal catalog beyond it being a folder on disk.

streaming-with-kafka-and-flink-guide
  -> This entire guide processed data in batches -- kiosko_orders_at_scale.csv already existed, complete, before the pipeline started. If Kiosko wanted a sales report updated in real time instead of a complete run, neither the DataFrame API in batch mode nor any code from this guide solves that by design.

data-reliability-and-governance-guide
  -> This capstone's pipeline has no formal lineage system beyond the comments in the code itself, no versioned data contract a consuming team could validate before using fact_orders_at_scale, and no granular access control -- who can see each product's real unit_cost, and who should only see aggregated revenue?

aws-core-services-guide
  -> This entire pipeline ran on local[*], on a laptop, at $0 cost and with no cloud account at all. The day lesson 6's Scenario 3 (16.4 TB, IT DEPENDS) becomes real for Kiosko, EMR or Databricks are the managed Spark that replaces local[*] with a real cluster -- never built here.

cost-optimization-caching-guide
  -> This module's lesson 6 closed the IT DEPENDS zone with an explicit instruction: 'measure the real cost of a cluster against a bigger node' -- but this entire guide never built that measurement. That is, specifically, this linked guide's FinOps discipline.

advanced-sql-querying-guide
  -> The .explain() you read in modules 5 and 6 is Catalyst's physical plan over distributed DataFrames -- a different topic from a traditional relational engine's (PostgreSQL, MySQL) EXPLAIN, with its own index tuning and execution plans.

Eight lines, eight real gaps, eight guides. None of these limitations is a bug in this pipeline — they are, exactly, the boundary this guide drew from its own DESIGN doc: "here you go as far as the DataFrame API, shuffle, joins, Catalyst, and partitioned Parquet in local mode; each thread's specific depth, opened and not closed, lives in its corresponding sibling guide."

Going deeper: one by one, the ecosystem's guides

airflow-and-declarative-orchestration-guide. Every script in this guide — including this module's lesson 3's, the most complete of all — you ran yourself, by hand, with python3 script_name.py. No system schedules it to run on its own every early morning, no system waits for a new kiosko_orders_at_scale.csv to exist before triggering the run, and no system alerts you if a complete run simply didn't happen. This is, precisely, the boundary this guide's DESIGN doc already explicitly named: "here the PySpark script runs by hand, from the terminal, in local mode — the exact moment that stops being enough gets named (M8) and points to that guide, without repeating its content." This sibling guide genuinely installs Airflow, with DAGs, sensors waiting for files, retries managed by the orchestrator itself — including, specifically, the spark-submit task as one more piece of a DAG — and platform-level observability of every run, not a single terminal process.

dbt-analytics-engineering-guide. fact_orders_at_scale, as you built it in this module's lesson 3, is the result of Python code living in a .py file, with no explicit versioning of its business rules as a declarative artifact, no dbt test-style tests someone with no PySpark experience could write or read, and no automatic lineage — the ability to answer, without opening the code, "exactly which source tables does this report's revenue column come from?" This sibling guide turns an equivalent transformation into a real dbt project, and specifically names the dbt-spark adapter: SQL models versioned in Git, reusable macros, automatic declarative tests, and a lineage graph anyone on the team can query, running over the same Spark engine you already master.

lakehouse-and-iceberg-guide. fact_orders_at_scale_m8.parquet, written with partitionBy("store_id") in lesson 3, is flat columnar Parquet, organized by folders: it has no version history (you can't ask "what did this dataset look like a week ago, before the last rebuild?"), doesn't support ACID transactions if two different processes tried writing to it at the same time, and there's no formal catalog tracking its existence beyond being a folder with .parquet files inside. This sibling guide builds the missing layer on top of exactly this kind of file: Apache Iceberg or Delta Lake, with real time travel, controlled schema evolution, and catalogs several different engines — including Spark, which you already know deeply — can query coherently and concurrently.

streaming-with-kafka-and-flink-guide. This entire guide, since module 1, processed data that already existed completely before any script started running — kiosko_orders_at_scale.csv got generated once, in module 4, and every later lesson read it as a finished file. This guide's module 7 named, in a paragraph with no code, that Structured Streaming reuses the same DataFrame API for continuous data; but neither that module nor any other built a single real streaming pipeline. This sibling guide teaches Kafka as a distributed messaging system, Flink for continuous processing, and CDC (Change Data Capture) to propagate changes without needing to wait for a complete file to finish writing.

data-reliability-and-governance-guide. This capstone's pipeline has no formal lineage system beyond the comments you left in your own code, no data contract as a versioned artifact a consuming team could validate before trusting fact_orders_at_scale, no organization-level data observability, and no granular access control — who on the team can see each product's real unit_cost in dim_product, and who should only see revenue aggregated by store? This sibling guide builds exactly those capabilities, at the scale of a complete organization running many pipelines, not a single capstone.

aws-core-services-guide. Every line of code in this entire guide ran with master("local[*]"), on your own laptop, at $0 cost and with no cloud account at all — exactly as this guide's DESIGN doc declared from the start: "this guide is 100% local, $0, local[*], no cloud or Databricks account." The day this module's lesson 6's Scenario 3 — 16.4 TB, IT DEPENDS verdict — stopped being hypothetical for Kiosko, this sibling guide teaches the concrete managed-Spark-in-the-cloud services (EMR, Glue) that replace local[*] with a real cluster, with real autoscaling and resource managers — the same PySpark code you wrote here, running unchanged over that infrastructure.

cost-optimization-caching-guide. This module's lesson 6 closed the IT DEPENDS zone with an explicit instruction: "measure the real cost of a cluster against a bigger node" — but this entire guide never built that measurement. How much does a managed Spark cluster with the capacity lesson 6's Scenario 3 needs cost, in dollars per month, compared to a single node with lots of RAM running DuckDB or Polars? This sibling guide — FinOps, specifically — teaches how to build exactly that kind of real cost comparison, the piece turning this guide's IT DEPENDS verdict into a concrete business decision.

advanced-sql-querying-guide (linked guide, from a different ecosystem). The .explain() you read in modules 5 and 6 — BroadcastHashJoin, Exchange, AdaptiveSparkPlan — is the Catalyst optimizer's physical plan over distributed DataFrames, a completely different mechanism from a traditional relational engine's EXPLAIN, like PostgreSQL or MySQL, with its own language of indexes, table statistics, and execution plans over data that never gets partitioned across several machines. This linked guide teaches that second discipline, complementary to but distinct from the one you mastered here.

Common mistakes

Feeling this pipeline "is useless" because it's missing these eight things. What happens: someone finishes this lesson feeling that everything built across this guide's eight modules was, at bottom, an incomplete exercise. Why it happens: seeing a list of eight gaps, all together, feels overwhelming, especially after lesson 6's honesty about whether Kiosko needed Spark. How to spot it: if your conclusion is "so I didn't really learn Spark," reread this lesson's analogy — the machine you learned to operate is genuine, even if you don't yet know how to run a data platform's whole factory. How to fix it: every pattern you mastered — the DataFrame API, shuffle, joins with real criteria, Catalyst, caching with real criteria, partitioned Parquet, pandas_udf, and the honest decision tree — is exactly what a data engineer needs, regardless of the scale of the pipeline they build afterward. What's missing isn't "the right pattern" — it's the team and organization infrastructure around that pattern, and that's precisely what each sibling guide teaches.

Trying to learn all eight sibling guides at once. What happens: someone, motivated by this lesson's complete map, tries opening all eight guides in parallel, without finishing any of them in depth. Why it happens: seeing eight gaps explicitly named creates the urge to "resolve them all now." How to spot it: if you have eight browser tabs open with different guides and haven't completed even the first module of any of them, that's a sign of scattering, not progress. How to fix it: choose one sibling guide, the one resolving the gap that matters most for your concrete situation — does your current job need scheduling real jobs? start with Airflow; does your team already use dbt for other transformations? start with dbt-analytics-engineering-guide — and finish it before opening the next one. This lesson's map is meant to orient you, not to be consumed in a single sitting.

Confusing "sibling guide within the Data Engineering ecosystem" with "linked guide from a different ecosystem," and treating them with the same priority. What happens: someone gives the same weight to advanced-sql-querying-guide (linked, from a different ecosystem, referenced because Spark's .explain() superficially resembles traditional SQL's EXPLAIN) as to airflow-and-declarative-orchestration-guide (sibling, continuing a thread explicitly opened within the Data Engineering ecosystem itself). Why it happens: this lesson's list presents both kinds of guide with no explicit visual distinction in ecosystem_map.py's output. How to spot it: if you can't say, without thinking hard, which of this lesson's eight guides belong to the same Data Engineering ecosystem as this guide and which come from a different one, review this lesson's "going deeper" section — only advanced-sql-querying-guide is explicitly marked as linked. How to fix it: the list's first seven guides continue threads this ecosystem's own DESIGN doc deliberately left open; the last one is borrowed from a different ecosystem, useful but lower priority if your goal is completing the Data Engineering ecosystem first.

Exercises

Exercise 1 — Map your own gap. Think of a limitation in this capstone's pipeline you yourself noticed across this guide's eight modules, one not explicitly named in this lesson's table. Identify which of the eight guides it would correspond to, and justify your choice in 2-3 sentences.

See solution

There's no single correct answer — the exercise asks for personal reflection — but a reasonable example: "this guide's pipeline never validates that order_ts falls within a reasonable date range, nor that quantity is always positive." That specific limitation would fall under data-reliability-and-governance-guide, if thought of as an organization-level data-quality rule that should apply consistently to any new pipeline, not just Kiosko's.

Exercise 2 — Tell a sibling guide apart from a linked guide. Without looking at this lesson's "going deeper" section again, classify each of these four guides as sibling (part of the Data Engineering ecosystem, continues a thread this guide opened) or linked (from a different ecosystem, referenced for an indirect reason):

  • (a) lakehouse-and-iceberg-guide
  • (b) advanced-sql-querying-guide
  • (c) aws-core-services-guide
  • (d) data-reliability-and-governance-guide
See solution
  • (a) Sibling. Goes deep on the lakehouse's table format, a thread this guide's module 7 explicitly opened with flat Parquet.
  • (b) Linked. Belongs to the SQL ecosystem, referenced because Spark's .explain() shares vocabulary ("execution plan") with a relational engine's EXPLAIN, but it's a different mechanism.
  • (c) Linked. Belongs to the cloud infrastructure ecosystem, referenced because this entire guide was deliberately local[*], with no cloud account.
  • (d) Sibling. Continues the data-governance thread, governing the same Parquet this capstone wrote.

Exercise 3 — Argue why this guide didn't try to teach all eight things at once. Using the market warning that opened this guide's DESIGN doc — the criticism that typical content "jumps straight from Python to Spark" with no criterion for when NOT to distribute — explain in 2-3 sentences why it would have been a pedagogical mistake for this guide to try to cover, even superficially, the eight sibling and linked guides' topics within its own eight modules.

See solution

If this guide had tried to touch orchestration, dbt, the lakehouse, streaming, data governance, cloud infrastructure, FinOps, and advanced SQL within its own eight modules, every topic would have gotten, at most, a superficial mention, without the real, verified code every concept it did cover in depth got (the DataFrame API, shuffle, joins with real criteria, Catalyst, partitioned Parquet, pandas_udf). Separating each topic into its own sibling guide, complete and terminal in itself, is what lets each one receive the same runnable treatment Spark got here — code genuinely run, with literal output, verified with assert — instead of a passing mention that would leave the reader with the illusion of having learned something they never actually ran.

Summary and next step

In this lesson you stepped back from the code and traced the ecosystem's complete map: seven sibling guides — orchestration, dbt-versioned transformation, the lakehouse with Iceberg, streaming with Kafka and Flink, data governance, cloud infrastructure, FinOps — each resolving a specific, named gap in this distributed pipeline, plus one linked guide — advanced SQL — for what was never this guide's responsibility. None of the limitations named here invalidate what you built — it's, exactly, the boundary this guide drew from its own design, now seen complete with the finished capstone in front of you.

Before closing out the guide you should be able to: name, from memory, at least four of this lesson's eight guides and what each one resolves; explain the difference between a sibling guide and a linked guide; and choose, with your own judgment — not by list order — which guide you'd follow first based on your own situation.

Lesson 8, the final mini-project closing out this entire guide, brings the complete distributed pipeline together one last time — assembled, verified, with every decision justified, and with the decision tree applied — and produces the final report: spark-and-distributed-processing-guide's definitive delivery.

Resources

  • Apache Airflow — official documentation, airflow-and-declarative-orchestration-guide's starting point. airflow.apache.org/docs.
  • dbt Labs — "What is dbt?", the official introduction to the project dbt-analytics-engineering-guide goes deep on, including the dbt-spark adapter. docs.getdbt.com/docs/introduction.
  • Apache Iceberg — official documentation, lakehouse-and-iceberg-guide's starting point over exactly the kind of Parquet this guide wrote. iceberg.apache.org/docs/latest.
  • Apache Spark — Structured Streaming Overview, the same quote already used in this guide's module 7, the foundation for the thread streaming-with-kafka-and-flink-guide picks back up. spark.apache.org/docs/latest/streaming/index.html.
  • This guide's DESIGN doc (spark-and-distributed-processing-guide/DISENO.md) — the "Place in the ecosystem" and "NOT in scope" sections, the exact source for the boundary this lesson traces. src/guides/spark-and-distributed-processing-guide/DISENO.md.
← PreviousThe decision tree: does Kiosko actually need Spark?Next →Project: Kiosko's first distributed pipeline