Module introduction: freshness, volume, and lineage

Why this module exists

Module 5's closing project ended with a number that hadn't changed since module 3: DIMENSIONS_STILL_OPEN = ["freshness"], printed, literally, as the final line of a report that already covered five of the six data quality dimensions module 1 defined. completeness, uniqueness, and validity — modules 1 and 2. consistency — module 3, with ORD-9508 caught by an anti-join. accuracy — module 5, with ORD-9509 finally flagged, deviation=49.0, closing the dollars-to-cents bug that opened this guide. Five tools, five dimensions, each built over data already loaded as rows of a Polars DataFrame.

Freshness never had that luck, and not for lack of evidence. Module 1's lesson 5 already confirmed it, with simple arithmetic over fixed dates: orders_2026-08-14.csv was supposed to be available for review by 2026-08-15 at the latest (the informal 24-hour SLA), and Kiosko's quality apparatus doesn't review it until 2026-08-16 at 09:00 — 33 hours after that deadline passed, 57 hours after the day the first delivery was expected. That calculation, however, stayed inside a module 1 lesson: prose, an example script, no reusable check_freshness() any future pipeline could import and call. This module finally writes that function — and confirms, with executed evidence on the same twelve rows as always, that it fails, with the exact hours of delay.

But this module doesn't stop there, for an important reason: freshness is the sixth and last of the six row-and-table dimensions this module closes, but it isn't the only thing missing from Kiosko's quality system. Two more questions remain completely unanswered after module 5, and neither one fits inside the six dimensions: does the file have a reasonable size? — neither suspiciously truncated, nor suspiciously inflated — and where, exactly, does every column of Kiosko's warehouse come from? — a question no schema, no contract, none of the five earlier tools tried to answer, because none of them was designed for that. This module builds all three pieces together: check_freshness(), check_volume(), and LINEAGE_MAP, this entire ecosystem's first complete lineage map.

Connection to the previous module. Module 5 said it, almost word for word, in its final line: "Module 6 [...] closes the complete diagnosis of the six dimensions [...]. And, beyond the six dimensions, that module maps Kiosko's lineage for the first time." This module fulfills that complete promise — freshness, the last pending piece, and two new capabilities none of the five earlier tools could offer.

An analogy ahead of time: the expiration date, not whether the milk exists

You're going to build freshness's complete analogy in lesson 2, but it's worth previewing the central idea, because it organizes this entire module: ask yourself what "is this milk fresh?" exactly means. It isn't the same as asking "does milk exist in the carton?" — any of the five earlier tools would confirm that, by checking whether the unit_price field is present, whether quantity is positive. Freshness asks something different: how much time passed since it was bottled, compared to the moment you're drinking it? That question doesn't live in any individual sip of milk — it lives in the date printed on the whole carton, compared against the kitchen clock. This module's lessons 2 and 3 build that idea with technical precision; for now, it's enough to notice it's a question of a different nature from the five this ecosystem already knows how to answer.

The map of this guide's 8 modules (reminder)

#	Module	What it's about
1	When green doesn't mean correct	The green checkmark lie; the six dimensions; diagnosing S04 without fixing anything.
2	Declarative data quality tests with Pandera	OrdersSchema, catching completeness/uniqueness/validity — three of six dimensions.
3	Consistency and referential checks	Referential integrity between tables; an anti-join that catches S04's orphan product_id.
4	Data contracts as versioned artifacts	What a data contract is; orders_contract.yaml; the contract generates module 2's tests.
5	Accuracy and deterministic anomaly detection	A price baseline; the dollars-to-cents bug, finally caught.
6	Freshness, volume, and lineage (you are here)	Freshness and volume as file-level properties; lineage mapped by hand.
7	The incident and data governance	Quarantine, alert, runbook; role-based access; PII masking.
8	Project: Kiosko's trust system	The capstone, run against S04 and against a clean day.

The map of this module

Lesson    Question it answers
────────  ──────────────────────────────────────────────────────────────
L1        (this one) Why freshness remained pending until now, and what
          two new capabilities -- volume and lineage -- join it.
L2        Why freshness is a whole-file property, never an individual
          row's -- with evidence, not just prose.
L3        Why any check's "now" has to be a fixed constant, never
          datetime.now() or the real clock.
L4        check_freshness(df, run_at=PIPELINE_RUN_AT, sla_hours=24) --
          written, run on real S04: FAILS, with the exact hours.
L5        check_volume(df, min_rows, max_rows) -- too few or too
          many rows -- run on S04: PASSES.
L6        What lineage answers that no contract answers: where every
          column comes from, not just what shape it should have.
L7        LINEAGE_MAP mapped by hand over Kiosko's warehouse;
          OpenLineage/Marquez named as the production version.
L8        Project: S04's complete freshness, volume, and lineage
          report.

Lessons 2 and 3 complete the conceptual argument before writing a single line of the central function: why freshness needs to be looked at differently (lesson 2), and why the time reference has to be an injected constant, never the system clock (lesson 3) — a design principle that matters well beyond this guide. Lesson 4 is this module's central moment: check_freshness(), really written and run, failing with the exact hours of delay over S04. Lesson 5 adds check_volume(), with the deliberate contrast that this one does pass — not everything in S04 is broken. Lessons 6 and 7 change topic entirely: from "is it correct?" to "where does it come from?" — the lineage question, closed with a hand-mapped map and the name of the standard the industry uses to automate it. And lesson 8, the project, assembles all three pieces into a single report.

Going deeper: why these three pieces live together in one module

It's worth asking, before moving on, why freshness, volume, and lineage — three capabilities that at first glance look so different from each other — share a single module, instead of splitting into three. The answer isn't an accident of organization: the three share something structural none of modules 1 through 5's five tools shares. validate_orders(), OrdersSchema, validate_referential_integrity(), module 4's contract, and check_price_baseline() all answer, one way or another, the question "is this row, compared against another row or against a rule, okay?" — their unit of work is always the individual row, even though some need to look at another table to answer (module 3) or an external baseline (module 5).

Freshness and volume break that pattern at the root: there's no row you can ask "are you fresh?" or "are you part of a correctly-sized file?" — those questions only make sense over the complete set. And lineage breaks the pattern even further: it doesn't even ask whether something "is okay" — it asks where it came from, a category of question entirely different from quality. All three, though, share the same level of abstraction: they look at the file or table as a whole unit, never an isolated row. Grouping them into one module isn't a convenience decision — it's recognizing that, after five modules looking at rows, this guide needs a complete change of altitude before it can close Kiosko's trust system.

The boundary: what does NOT enter this module

Two concrete boundaries, worth naming from the start because they're going to reappear in lessons 6 and 7. First: this module names OpenLineage and Marquez — the open lineage standard, and its reference implementation — but installs no server, runs no real-time collector from Airflow, Spark, or dbt. LINEAGE_MAP is a Python dictionary, hand-written, running on the same laptop as always, at $0 cost — exactly the same discipline that has sustained this ecosystem's eight earlier guides. Actually running OpenLineage/Marquez, with events coming in real time from an orchestrator, is airflow-and-declarative-orchestration-guide's territory.

Second: dbt docs generate already builds an automatic lineage graph — but it's the lineage of a single dbt project, the dependencies between .sql models inside that project. The lineage this module builds crosses tools: from a raw CSV, to a DuckDB table, to a calculated warehouse column — a path no dbt project, on its own, can fully document, because it starts before dbt even enters the picture. Lesson 6 returns to this distinction with precision.

Common mistakes

Confusing "volume" with a seventh data quality dimension. What happens: someone, after finishing this module, starts talking about "the seven data quality dimensions," adding volume to the completeness/uniqueness/validity/consistency/freshness/accuracy list module 1 already established. Why it happens: check_volume() gets built in this module, with the same care as any of the six earlier checks, so it feels like it belongs to the same category. How to spot it: review module 1's exact definition — the six dimensions describe properties of the data's content (is it complete? is it unique? is it valid? is it consistent across tables? is it recent? is it correct?). Volume describes something different: how many physical rows the file brings, saying nothing about whether those rows, individually, are okay. How to fix it: treat check_volume() as an additional structural check, at the same level as freshness (both are properties of the whole file, not of a row), but outside the canonical list of six dimensions this ecosystem already closed since module 1.

Expecting this module to automatically fix the five failures you already know about in S04. What happens: someone, familiar with modules 2, 3, and 5's pattern — each catching one more dimension —, expects this module to also reduce S04's broken-row count. Why it happens: earlier modules always ended with a number going down (from "0 dimensions covered" to "5 of 6"). How to spot it: if your expectation is that S04 ends this module with fewer than six flagged rows, review exactly what freshness does — it doesn't flag any new row, it flags the whole file as late, with no change to the broken-row count module 5's project already closed at six. How to fix it: the unit that changes in this module isn't "how many rows are wrong" — that already got fixed at six since module 5's project —, it's "how many properties of the whole system already have a runnable check": freshness, volume, and now lineage too.

Exercises

Exercise 1 — Recall, from memory, the exact hours module 1 already calculated. Without going back to that lesson, try to remember: how many hours passed between the 24-hour SLA's deadline and PIPELINE_RUN_AT? And between the expected arrival day and PIPELINE_RUN_AT? Confirm your answer against module 1's lesson 5.

See solution

33.0 hours past SLA_DEADLINE (2026-08-15T00:00:00), and 57.0 hours since EXPECTED_ARRIVAL (2026-08-14T00:00:00), both measured against PIPELINE_RUN_AT (2026-08-16T09:00:00). It's worth noting, before lesson 4, that those two numbers used the calendar day the file was expected as their reference — this module's lesson 4 measures freshness in a slightly different, more precise way: against the most recent order_ts that actually exists inside the file, not against an abstractly expected date. The two approaches are consistent with each other, but they don't produce exactly the same hour count — pay attention to that difference when you reach lesson 4.

Exercise 2 — Name, without looking back, the five tools that already exist in this guide and which dimension each one catches. From memory, complete: validate_orders() / OrdersSchema → ___; validate_referential_integrity() → ___; check_price_baseline() → ___.

See solution

validate_orders() (module 1) and OrdersSchema (module 2, regenerated from the contract in module 4) → completeness, uniqueness, validity. validate_referential_integrity() (module 3) → consistency. check_price_baseline() (module 5) → accuracy. With this module's check_freshness(), the six dimensions module 1 defined finally each have their own runnable tool.

Exercise 3 — Predict, before lesson 6, what specific question none of the five already-built tools can answer. In 1-2 sentences, without having read lesson 6 yet, try to anticipate: what question about Kiosko's data can neither OrdersSchema, nor module 4's contract, nor check_price_baseline() answer?

See solution

There's no single "correct" answer before reading lesson 6, but a reasonable answer points to this: none of the five tools says where a value comes from — all of them compare a value already present against a rule (is it null? is it unique? does it exist in another table? does it look like the reference price?), but none traces a column's complete path, from the raw CSV file to the warehouse's final table, through any intermediate calculation (like revenue, which doesn't even exist yet in orders_s04 — it gets calculated later). That's, precisely, the question lesson 6 opens with.

Summary and next step

In this lesson you learned module 6's complete map: why freshness stayed pending from module 1 until now, and why two new capabilities — volume and lineage — join that last dimension instead of appearing in separate modules. You saw the complete map of the eight lessons that follow, and you drew, from the start, the boundary with OpenLineage/Marquez in production and with dbt docs's automatic lineage.

Before moving on you should be able to: explain why volume isn't a seventh data quality dimension; name the five tools that already exist in this guide and which dimension each one catches; and anticipate, in your own words, what kind of question lineage answers that no earlier tool can answer.

Lesson 2 starts the real work: it builds, with executed evidence, the argument for why freshness needs to be looked at as a whole-file property — never a row's — before writing a single line of check_freshness().

Resources

• Module 1, lesson 5, of this same guide ("Meet S04: Kiosko's fourth store") — the source of the 33.0 and 57.0 hour numbers recalled in this lesson. src/guides/data-reliability-and-governance-guide/workbook/module-01-when-green-does-not-mean-correct/en/05-meet-s04-kioskos-fourth-store.md. In English.
• Module 5, project (lesson 8), of this same guide — the source of DIMENSIONS_STILL_OPEN = ["freshness"], the exact point this module picks back up. src/guides/data-reliability-and-governance-guide/workbook/module-05-accuracy-and-deterministic-anomaly-detection/en/08-project-s04s-accuracy-audit.md. In English.
• OpenLineage — official documentation (the open standard this module names, without installing, in lessons 6 and 7). openlineage.io/docs. In English.
• Marquez — project site (OpenLineage's reference implementation). marquezproject.ai. In English.
• This guide's DESIGN — this module 6's exact mandate. src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.