What lineage answers that a contract does not

Description

With check_freshness() and check_volume() complete, this guide already has a runnable function for each of the six row-and-table dimensions, plus the volume check. This lesson changes topic entirely: instead of asking "is this data okay?" — the question that has organized every lesson since module 1 — it asks something no earlier tool in this guide tried to answer: where, exactly, does every column in Kiosko's warehouse come from?

Connection to the module. This lesson is purely conceptual — with no dictionary written yet, that's lesson 7 — but it's essential before building it: if it isn't clear what question lineage answers that module 4's contract doesn't, LINEAGE_MAP would feel like redundant work instead of a genuinely new layer.

A complete analogy: a column's family tree

A birth certificate describes a person: name, date, place. It's a valid document, complete for what it sets out to document. But it says nothing about where that person's traits come from — which grandparent contributed the surname, which family line a certain feature comes from. For that you need a different document: a family tree, which traces, generation by generation, where each branch descends from.

A data contract — like module 4's orders_contract.yaml — is a column's birth certificate: it says unit_price must be a float, non-null, >= 0. It's a valid, complete description of what that column should be. But it says nothing about where it comes from — whether unit_price got copied directly from a point-of-sale system, whether it went through some currency conversion, or whether another column, in another table, got calculated from it. Lineage is the family tree: it doesn't describe what a column should look like, it describes its ancestry — which column, in which table, at which point of the pipeline, it descends from.

Worked example: what S04's contract never declared

Module 4's contract, orders_contract.yaml, declared rules for three columns: order_id, unit_price, quantity. It's worth confirming, with a direct comparison, how much of Kiosko's reality falls outside that contract — not because it's poorly written, but because it never set out to cover this.

# contract_vs_warehouse.py
ORDERS_S04_COLUMNS = ["order_id", "store_id", "product_id", "quantity", "unit_price", "order_ts"]
CONTRACT_SCHEMA_COLUMNS = ["order_id", "unit_price", "quantity"]
FACT_ORDERS_COLUMNS = ["order_id", "store_id", "product_id", "quantity", "unit_price", "revenue", "order_ts"]

no_contract_rule = sorted(set(ORDERS_S04_COLUMNS) - set(CONTRACT_SCHEMA_COLUMNS))
print(f"orders_s04 columns: {ORDERS_S04_COLUMNS}")
print(f"Columns with rules in the contract: {CONTRACT_SCHEMA_COLUMNS}")
print(f"orders_s04 columns WITHOUT any rule in the contract: {no_contract_rule}")
print()

new_in_fact = sorted(set(FACT_ORDERS_COLUMNS) - set(ORDERS_S04_COLUMNS))
print(f"fact_orders columns: {FACT_ORDERS_COLUMNS}")
print(f"fact_orders columns that do NOT exist in orders_s04 (or in the contract): {new_in_fact}")

What to expect. Running python3 contract_vs_warehouse.py, the output is exactly this:

orders_s04 columns: ['order_id', 'store_id', 'product_id', 'quantity', 'unit_price', 'order_ts']
Columns with rules in the contract: ['order_id', 'unit_price', 'quantity']
orders_s04 columns WITHOUT any rule in the contract: ['order_ts', 'product_id', 'store_id']

fact_orders columns: ['order_id', 'store_id', 'product_id', 'quantity', 'unit_price', 'revenue', 'order_ts']
fact_orders columns that do NOT exist in orders_s04 (or in the contract): ['revenue']

Two findings, each with a different reading. First: orders_s04 has six real columns (confirmed by DESCRIBE orders_s04 since module 2), but the contract only declared rules for three — store_id, product_id, and order_ts never had a Pandera Field or an entry under the YAML's schema:. That isn't a flaw in the contract — it covered exactly what module 4's lesson 3 identified as necessary for the business rules already known — but it confirms that not even within the columns that do exist in the original file does the contract claim to be a complete catalog.

Second, and more important for this lesson: fact_orders — the warehouse's final table, the one data-modeling-for-analytics-guide built — has a column, revenue, that doesn't exist anywhere inside orders_s04. Nobody sends it in the CSV. No Field in the contract mentions it, because it makes no sense to declare a rule about a column that doesn't even exist yet at the point where the contract lives. revenue gets calculated, at some step between orders_s04 and fact_orders, from other columns — and neither the contract, nor OrdersSchema, nor any of this guide's five earlier tools says, anywhere, which ones.

Comparison table: what each artifact answers

Question	Does the contract (module 4) answer it?	Does lineage (this module) answer it?
Can unit_price be null?	Yes — nullable: false	Not its question
How many rows should the file bring?	Yes — sla.row_count	Not its question
How late can it arrive?	Yes — sla.freshness_hours	Not its question
Which column, from which table, does fact_orders.revenue come from?	No — never declared it	Yes — exactly the question it answers
Is dim_store.country copied data or calculated data?	No — country doesn't even live in orders_s04	Yes — lineage also traces transformations, not just copies
If unit_price changes meaning (dollars to cents), which other warehouse columns are affected?	No — the contract describes one table at a time	Yes — lineage crosses tables and shows downstream impact

The table's last row deserves a pause. The dollars-to-cents bug module 5 caught (ORD-9509, unit_price=60.00) lived inside orders_s04, a single table. But if that same kind of error had gone unnoticed and reached fact_orders, it wouldn't stay contained in one column: revenue — calculated from unit_price — would also be wrong, and any report summing revenue (like the one data-modeling-for-analytics-guide already built) would inherit the error with nobody knowing where it came from. With no lineage map, tracing that impact would mean reading, by hand, every SQL query in the warehouse looking for where unit_price gets used — exactly the work an already-written lineage map avoids.

Diagram: two documents, two different questions about the same column

flowchart LR
    subgraph CONTRACT["orders_contract.yaml (M4)"]
        C1["unit_price:\ntype=float\nnullable=false\nminimum=0"]
    end
    subgraph LINEAGE["LINEAGE_MAP (M6, lesson 7)"]
        L1["fact_orders.revenue\n<- orders.quantity\n<- orders.unit_price"]
    end
    C1 -.->|"'what shape should\nthis value have?'"| Q1["The contract's question"]
    L1 -.->|"'which column does\nthis value come from,\nand which other columns\ninherit it?'"| Q2["Lineage's question"]

Going deeper: why lineage matters more the bigger the ecosystem grows

It's worth connecting this lesson to NIEVA Data Engineering's complete ecosystem map. Kiosko already crossed, in the eight guides before this one, several tool boundaries: a raw CSV (data-engineering-foundations-guide), a warehouse modeled in DuckDB (data-modeling-for-analytics-guide), a dbt project transforming those tables (dbt-analytics-engineering-guide), and — named, though not built in depth in this guide — Spark and Iceberg tables. Every tool crossing is a point where lineage can get lost if nobody documents it explicitly: dbt docs generate knows how to trace the dependency graph inside a dbt project (which .sql model depends on which other), but it knows nothing about which column of the original CSV fed that project's first table — that first link lives outside dbt's scope, in the extraction and ingestion step.

This is, precisely, the boundary this module's design drew from the start: a single tool's automatic lineage (dbt docs) documents one stretch of the complete path. This lesson's lineage — and the one lesson 7 builds — crosses tools: from the CSV, through DuckDB, to the warehouse's calculated column. The more different tools that participate in a real pipeline (and NIEVA's complete ecosystem has already shown there are several), the more valuable it becomes to have a map that crosses all of them, instead of trusting that each tool perfectly documents its own stretch and that those stretches, together, tell the complete story with no gaps.

Common mistakes

Thinking lineage replaces the contract, or that one of the two becomes unnecessary if the other exists. What happens: someone, after this lesson, concludes that if a complete lineage map already exists, module 4's contract becomes redundant — or the other way around. Why it happens: both artifacts describe "where trust in Kiosko's data comes from," and it's easy to think they serve the same role. How to spot it: review this lesson's comparison table — every row shows a question only one of the two artifacts answers. No row has a "Yes" in both columns. How to fix it: treat them as complementary, never substitutes: the contract certifies a table's shape at a given moment; lineage traces how that table, and each of its columns, came to exist. A complete trust system — the one module 8's project assembles — needs both.

Assuming lineage only matters for calculated columns, never for directly copied ones. What happens: someone, seeing the revenue example (a genuinely calculated column), concludes lineage only makes sense for complex transformations, and that a 1:1 copied column (like dim_product.product_name, taken directly from products.product_name) needs no mapping at all. Why it happens: a direct copy feels "obvious," with no mystery to document. How to spot it: ask yourself what would happen if, someday, someone renamed product_name in the original source — with no lineage map documenting the relationship, nobody would automatically know dim_product.product_name also needs updating. How to fix it: lesson 7 is going to map every warehouse column, copied or calculated alike — the distinction between "direct copy" and "transformation" is useful information within the map (and lesson 7 explicitly flags it for dim_store.country's case), but it isn't a reason to omit copied columns from the complete map.

Believing lineage is exclusive to large systems, and not worth it for a case as small as Kiosko's. What happens: someone argues that, with only three tables (orders_s04, dim_product, dim_store) and fifteen columns total, documenting lineage by hand is unnecessary work for a case this size. Why it happens: lineage sounds like a "big data" discipline — enormous graphs, hundreds of tables, something that only matters at scale. How to spot it: if your reasoning is "this is too small to need lineage," check how long it would take, with no map at all, to answer "what breaks if I change unit_price?" for Kiosko's real case — the answer is already in this lesson's Going deeper section: reviewing every SQL query in the warehouse by hand. How to fix it: lineage's value doesn't depend on the system's size, it depends on whether someone needs to answer "where does this come from?" or "what gets affected if this changes?" — questions that are already real for Kiosko with just three tables, and that become impossible to answer by hand, no matter the size, if nobody documents them from the start.

Exercises

Exercise 1 — Reproduce the column comparison in your own words. Without looking at this lesson's code, write from memory the list of orders_s04's six columns and the three that do have rules in the contract. Confirm your answer against contract_vs_warehouse.py.

See solution

orders_s04: order_id, store_id, product_id, quantity, unit_price, order_ts (confirmed by DESCRIBE orders_s04 in module 2, lesson 4). With rules in the contract: order_id (unique, non-null), unit_price (non-null, >= 0), quantity (non-null, > 0) — the same three OrdersSchema already declared since module 2. The other three (store_id, product_id, order_ts) exist in the table, but never had a Pandera rule or a YAML contract entry — not because they don't matter, but because no earlier lesson's tool yet needed an explicit rule about them (product_id, for example, gets validated another way: module 3's anti-join against dim_product, not a type or range rule).

Exercise 2 — Argue whether store_id would someday need an entry in LINEAGE_MAP even though it never had a rule in the contract. In 2-3 sentences, considering lineage and the contract answer different questions (confirmed by this lesson's comparison table), argue whether store_id should appear in lesson 7's lineage map, even though the contract never declared any rule about it.

See solution

Yes, it should appear — the absence of a rule in the contract says nothing about whether a column needs tracing in lineage, because they're completely different questions (confirmed by this lesson's comparison table: "what shape should it have?" versus "where does it come from?"). store_id does travel from orders_s04 all the way to fact_orders with no change, and it also participates in the join against dim_store that builds the complete star schema (data-modeling-for-analytics-guide) — anyone who needs to understand where a fact_orders row's store_id comes from needs that answer, regardless of whether the contract ever declared a validation rule about it.

Exercise 3 — Predict, before lesson 7, whether dim_store.country will have a "direct" or "derived" lineage mapping. Using what you already know from this guide's module 1 (the deterministic rule Bogotá→Colombia, Lima→Peru, Santiago→Chile, Mexico City→Mexico), predict whether LINEAGE_MAP["dim_store.country"] is going to point to a column that gets copied as-is, or one that gets transformed.

See solution

Transformed — country never exists as an independent column in any of Kiosko's raw sources (stores only has store_id, store_name, city, according to data-engineering-foundations-guide's DESIGN); it gets derived from city through a deterministic function, the same one lakehouse-and-iceberg-guide (module 4) already applied and this guide extended to S04 in its first lesson. This means, unlike a column like dim_product.product_name (copied with no change at all from products.product_name), dim_store.country does depend on transformation logic — even if simple and deterministic —, and a complete lineage map should, ideally, be able to distinguish between both kinds of relationship. Lesson 7 confirms this prediction with the real map.

Summary and next step

In this lesson you established, with a really-executed comparison (not just prose), the exact boundary between what a data contract answers and what lineage answers: module 4's contract declared rules for three of orders_s04's six columns, and no rule at all about revenue — a column that doesn't even exist until reaching fact_orders. You confirmed, with the family tree analogy, that lineage doesn't describe what a column should look like, but where it descends from — a question of a completely different nature, one that becomes more urgent the more different tools a real pipeline crosses.

Before moving on you should be able to: name, without looking at the comparison table again, at least two questions only the contract answers and two only lineage answers; and explain why revenue is this guide's clearest example of a column the contract could never have covered.

You have the complete conceptual argument. Lesson 7 turns it into code: LINEAGE_MAP, a Python dictionary mapped by hand over fact_orders, dim_product, and dim_store's real columns — and the name of the open standard, OpenLineage, the industry uses to automate exactly this map in production.

Resources

• Module 2, lesson 4, of this same guide — the source of DESCRIBE orders_s04, the table's six real columns compared in this lesson. src/guides/data-reliability-and-governance-guide/workbook/module-02-declarative-data-quality-tests-with-pandera/en/04-installing-pandera-and-bridging-duckdb-to-polars.md. In English.
• Module 4, lesson 3, of this same guide — the source of the three columns with rules in orders_contract.yaml, contrasted against the table's total. src/guides/data-reliability-and-governance-guide/workbook/module-04-data-contracts-as-versioned-artifacts/en/03-writing-orders-contract-yaml.md. In English.
• data-modeling-for-analytics-guide, module 8, project — the source of fact_orders's revenue column, calculated from quantity and unit_price. src/guides/data-modeling-for-analytics-guide/workbook/module-08-project-kioskos-analytics-warehouse/en/08-project-kioskos-first-analytics-warehouse.md. In English.
• OpenLineage — official documentation (the open standard lesson 7 names, with its Dataset/Job/Run model). openlineage.io/docs. In English.
• This guide's DESIGN — the exact boundary between this guide's lineage and a single dbt project's automatic dbt docs lineage. src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.