Mapping Kiosko's lineage by hand

Description

This lesson builds LINEAGE_MAP: a Python dictionary, hand-written, that traces every column of Kiosko's warehouse — fact_orders, dim_product, dim_store — back to the source column it descends from. It's the runnable answer to the question lesson 6 left open: where, exactly, does every piece of data Kiosko already built across eight earlier guides in this ecosystem come from. And it closes by naming OpenLineage and Marquez, the open standard the industry uses to automate this exact same map, with no server installed in this guide.

Connection to the module. Lesson 6 established, with evidence, that neither the contract nor any of the five earlier tools documents where a column comes from. This lesson fills that gap with this entire ecosystem's first complete lineage artifact — nine guides, and none earlier needed, until now, to answer this question explicitly.

An analogy: the genealogist who finally draws the complete tree

Lesson 6 introduced the family tree idea. This lesson is the moment a genealogist, after gathering birth certificates, parish records, and family interviews, finally sits down to draw the complete tree: every name, connected by a line to its direct ancestry. They don't invent any relationship — every line in the tree has to correspond to a real fact, verifiable in some document — but they also don't wait for an automatic system to draw it for them. With the information they already have, they trace it by hand.

LINEAGE_MAP is exactly that tree, drawn by hand for the first time. Every dictionary entry is a line in the tree: "this warehouse column descends from this other one, in this source table." No relationship gets invented — each one is already confirmed by this ecosystem's eight earlier guides' real work —, but nobody had put them together, until this lesson, into a single queryable document.

Worked example: LINEAGE_MAP, built and queried

Step 1 — the complete dictionary

# lineage.py
import polars as pl

LINEAGE_MAP = {
    "fact_orders.order_id": ["orders.order_id"],
    "fact_orders.store_id": ["orders.store_id"],
    "fact_orders.product_id": ["orders.product_id"],
    "fact_orders.quantity": ["orders.quantity"],
    "fact_orders.unit_price": ["orders.unit_price"],
    "fact_orders.revenue": ["orders.quantity", "orders.unit_price"],
    "fact_orders.order_ts": ["orders.order_ts"],
    "dim_product.product_id": ["products.product_id"],
    "dim_product.product_name": ["products.product_name"],
    "dim_product.category": ["products.category"],
    "dim_product.unit_cost": ["products.unit_cost"],
    "dim_store.store_id": ["stores.store_id"],
    "dim_store.store_name": ["stores.store_name"],
    "dim_store.city": ["stores.city"],
    "dim_store.country": ["stores.city"],
}

Fifteen entries, each shaped "table.derived_column": ["table.source_column", ...] — a list, not a single value, because a derived column can depend on several source columns at once. Notice two entries that deserve special attention. fact_orders.revenue points to two source columns (orders.quantity and orders.unit_price) — it confirms, right in the map, what lesson 6 already previewed: revenue isn't a copy of anything, it's the result of multiplying two columns that do exist in the source. And dim_store.country points to stores.city — a single source column, but not the same column (country doesn't exist in stores), but one transformed by the deterministic function lakehouse-and-iceberg-guide already used since its module 4.

Step 2 — query a specific column's lineage

# lineage.py -- continuation
def trace_column(column: str, lineage_map: dict[str, list[str]]) -> list[str]:
    """Returns a derived column's source columns, or [] if it isn't mapped."""
    return lineage_map.get(column, [])


if __name__ == "__main__":
    print(f"LINEAGE_MAP has {len(LINEAGE_MAP)} derived columns mapped\n")

    print("=== trace_column('fact_orders.revenue', LINEAGE_MAP) ===")
    print(trace_column("fact_orders.revenue", LINEAGE_MAP))

    print("\n=== trace_column('dim_store.country', LINEAGE_MAP) ===")
    print(trace_column("dim_store.country", LINEAGE_MAP))

    print("\n=== trace_column('fact_orders.unknown_column', LINEAGE_MAP) ===")
    print(trace_column("fact_orders.unknown_column", LINEAGE_MAP))

What to expect.

LINEAGE_MAP has 15 derived columns mapped

=== trace_column('fact_orders.revenue', LINEAGE_MAP) ===
['orders.quantity', 'orders.unit_price']

=== trace_column('dim_store.country', LINEAGE_MAP) ===
['stores.city']

=== trace_column('fact_orders.unknown_column', LINEAGE_MAP) ===
[]

trace_column() is deliberately simple: a .get() with an empty-list default, not one line more. But that simplicity is exactly the point — it answers, in a single call, a question that, without this map, would require reading every earlier guide's source code in this ecosystem looking for where each column gets calculated. The third case — a column that isn't in the map — returns [], not an exception: a query about a column with no documented lineage is valid information in itself (probably means that column is still missing this work), not a program error.

Step 3 — the complete report, as a table

# lineage.py -- continuation
def lineage_report(lineage_map: dict[str, list[str]]) -> pl.DataFrame:
    """Flattens LINEAGE_MAP into a table: derived column, source columns, how many."""
    rows = [
        {
            "derived_column": target,
            "source_columns": ", ".join(sources),
            "n_sources": len(sources),
        }
        for target, sources in lineage_map.items()
    ]
    return pl.DataFrame(rows).sort("derived_column")


if __name__ == "__main__":
    # ... continuation of the earlier block
    pl.Config.set_tbl_rows(20)
    pl.Config.set_fmt_str_lengths(40)  # so 'orders.quantity, orders.unit_price' doesn't get truncated
    print("\n=== lineage_report(LINEAGE_MAP) ===")
    print(lineage_report(LINEAGE_MAP))

What to expect.

=== lineage_report(LINEAGE_MAP) ===
shape: (15, 3)
┌──────────────────────────┬────────────────────────────────────┬───────────┐
│ derived_column           ┆ source_columns                     ┆ n_sources │
│ ---                      ┆ ---                                ┆ ---       │
│ str                      ┆ str                                ┆ i64       │
╞══════════════════════════╪════════════════════════════════════╪═══════════╡
│ dim_product.category     ┆ products.category                  ┆ 1         │
│ dim_product.product_id   ┆ products.product_id                ┆ 1         │
│ dim_product.product_name ┆ products.product_name              ┆ 1         │
│ dim_product.unit_cost    ┆ products.unit_cost                 ┆ 1         │
│ dim_store.city           ┆ stores.city                        ┆ 1         │
│ dim_store.country        ┆ stores.city                        ┆ 1         │
│ dim_store.store_id       ┆ stores.store_id                    ┆ 1         │
│ dim_store.store_name     ┆ stores.store_name                  ┆ 1         │
│ fact_orders.order_id     ┆ orders.order_id                    ┆ 1         │
│ fact_orders.order_ts     ┆ orders.order_ts                    ┆ 1         │
│ fact_orders.product_id   ┆ orders.product_id                  ┆ 1         │
│ fact_orders.quantity     ┆ orders.quantity                    ┆ 1         │
│ fact_orders.revenue      ┆ orders.quantity, orders.unit_price ┆ 2         │
│ fact_orders.store_id     ┆ orders.store_id                    ┆ 1         │
│ fact_orders.unit_price   ┆ orders.unit_price                  ┆ 1         │
└──────────────────────────┴────────────────────────────────────┴───────────┘

Fifteen rows, sorted alphabetically by derived_column, with n_sources as the column that most quickly highlights what's unusual: fourteen entries with n_sources=1 (a direct copy or a simple transformation of a single source column), and just one with n_sources=2 — fact_orders.revenue, the only column in Kiosko's entire warehouse combining two different sources. This report, run once, immediately answers a question this entire module has been building toward: of the warehouse's fifteen columns, which are simple and which need more than one piece of source data to exist?

Diagram: Kiosko's complete lineage tree

flowchart LR
    subgraph RAW["Raw sources"]
        O["orders\norder_id, store_id, product_id,\nquantity, unit_price, order_ts"]
        P["products\nproduct_id, product_name,\ncategory, unit_cost"]
        S["stores\nstore_id, store_name, city"]
    end
    subgraph WAREHOUSE["Warehouse (data-modeling-for-analytics-guide)"]
        FO["fact_orders\norder_id, store_id, product_id,\nquantity, unit_price, revenue, order_ts"]
        DP["dim_product\nproduct_id, product_name,\ncategory, unit_cost"]
        DS["dim_store\nstore_id, store_name,\ncity, country"]
    end

    O -->|"direct copy\n(5 columns)"| FO
    O -->|"quantity x unit_price"| FO
    P -->|"direct copy\n(4 columns)"| DP
    S -->|"direct copy\n(3 columns)"| DS
    S -->|"deterministic function\ncity -> country"| DS

Naming OpenLineage and Marquez: this same map's production version

LINEAGE_MAP works for Kiosko — fifteen columns, three tables, a dictionary that fits on one screen. It wouldn't work as well for a company with hundreds of tables and dozens of pipelines running in parallel: keeping that map by hand, updated every time someone adds a column or changes a transformation, would, at some point, become more work than the problem it solves. That's where OpenLineage comes in: an open standard — governed as a graduated project of the LF AI & Data Foundation, the same foundation that hosts projects like ONNX and Milvus — that defines a common model for any tool (Airflow, Spark, Flink, dbt, among others) to automatically emit lineage events, every time a real job runs, with no human having to write LINEAGE_MAP's equivalent by hand.

OpenLineage's model is organized around three concepts, with names that should already sound familiar after this lesson: Dataset (a table or file, like fact_orders or orders_s04), Job (a computation unit that transforms data, like the step that calculates revenue), and Run (a specific execution of that Job, at a given moment). Each of those three concepts can be enriched with facets — additional metadata defined by whoever uses them, like the column schema or quality statistics — without breaking compatibility with other tools reading the same event. Marquez, the project that started as open source from WeWork, is OpenLineage's reference implementation: a service that receives these events in real time through a standard-compatible endpoint, stores them, and draws the resulting lineage graph as a navigable visual map.

Neither tool gets installed in this guide — the $0, all-local rule that sustains this ecosystem's nine guides holds. But it's worth seeing, even just as an illustrative sketch, never run, how similar the underlying idea is to what you already built:

# ILLUSTRATIVE -- never runs or gets sent to any server in this guide.
# A real OpenLineage event has many more fields than this sketch;
# this only shows the conceptual similarity to LINEAGE_MAP.
openlineage_style_event = {
    "job": {"namespace": "kiosko", "name": "compute_fact_orders_revenue"},
    "run": {"runId": "some-uuid-generated-in-production"},
    "inputs": [
        {"namespace": "kiosko", "name": "orders", "facets": {"columns": ["quantity", "unit_price"]}}
    ],
    "outputs": [
        {"namespace": "kiosko", "name": "fact_orders", "facets": {"columns": ["revenue"]}}
    ],
}

The underlying difference isn't conceptual — both documents say, basically, the same thing: fact_orders.revenue depends on orders.quantity and orders.unit_price. The difference is who writes it and when. LINEAGE_MAP got written by a person, by hand, once, and someone has to remember to update it if something changes. An OpenLineage event gets emitted automatically by the tool that ran the Job — Airflow, Spark, dbt — every time it runs, with nobody having to remember anything. That automation is, precisely, what makes the standard worth it at a real company's scale — and it's also, exactly, what this guide chooses not to build, to stay local and free.

Going deeper: why hand-tracing has a clear limit, and where it sits

It's worth being honest about LINEAGE_MAP's limitations, not just its merits. Every one of its fifteen entries depends on a person — whoever wrote this lesson, in this case — already knowing the exact relationship between columns, with enough precision to write it with no ambiguity. That worked because this ecosystem's eight earlier guides already documented, with executed evidence, exactly how every table in Kiosko's warehouse gets built. In a real system, with pipelines written by different teams, at different times, with no central guide documenting every step, keeping a map like this by hand would become, over time, just as unreliable as the informal SLA that existed for S04 before module 4's contract — someone knows it by heart, until that person leaves or the system grows beyond what anyone's memory can hold.

That's, precisely, the business reason behind OpenLineage: it doesn't replace the reasoning you already did in this lesson — you still need to understand what relationships exist —, it replaces the mechanical work of keeping them written and updated, delegating it to the tools that already run the pipelines. The map you built in this lesson is correct and useful for Kiosko's size. It's also, exactly, the kind of manual work a real production system would automate as soon as the number of tables grew beyond what one person can keep in their head — the same argument this guide already made about S04's informal SLA before a versioned contract existed.

Common mistakes

Writing LINEAGE_MAP with the keys reversed: source as key, derived as value. What happens: someone builds the dictionary backward — {"orders.unit_price": ["fact_orders.revenue"]} instead of {"fact_orders.revenue": ["orders.unit_price", ...]} —, especially if they think of data flow "forward" (from source to destination) instead of "backward" (from destination to source). Why it happens: data, in reality, does flow from orders to fact_orders — thinking in that direction feels natural. How to spot it: if trace_column("fact_orders.revenue", LINEAGE_MAP) returns [] instead of the expected list, check which key you used. How to fix it: LINEAGE_MAP specifically answers the question "where does this column I already have come from?" — the question of an analyst looking at fact_orders and wondering about its origin — so the key is always the derived column (the destination), and the value is the list of source columns. It's the same direction as a family tree: you start from the person in front of you, and trace backward, toward their ancestors.

Assuming a column with n_sources=1 in lineage_report() is always an identical copy, with no transformation. What happens: someone sees dim_store.country with n_sources=1 in this lesson's report, and assumes that means "copied with no change at all," just like dim_store.city. Why it happens: n_sources counts how many source columns participate, not what kind of relationship exists between them — a single-column transformation (like city → country) has the same n_sources=1 as a direct copy. How to spot it: check the source column's name against the derived column's — if they're literally the same column name (city → city), it's a copy; if the name changes (city → country), there's a transformation in between, even though the source count is just as simple. How to fix it: don't confuse n_sources (how many columns participate) with the relationship type (copy versus transformation) — this lesson didn't include an explicit third column for that distinction in LINEAGE_MAP, so, for now, you need to read the names carefully; this lesson's Exercise 3 explores how to extend the map to make that distinction explicit.

Thinking this lesson's illustrative OpenLineage sketch is functional code you could copy and run. What happens: someone copies this lesson's openlineage_style_event dictionary and tries sending it to some endpoint, expecting it to work like a real OpenLineage event. Why it happens: the code block looks like any other runnable example in this guide. How to spot it: reread the comment heading that block — "ILLUSTRATIVE -- never runs or gets sent to any server in this guide" — and compare it against any other code block in this guide, which always comes followed by a "What to expect" with real output. This block has no "What to expect" at all, on purpose. How to fix it: treat that fragment for what it is — a demonstration of conceptual similarity, not a real implementation of the OpenLineage standard (which has many more required fields and a complete OpenAPI specification). For real code, consult OpenLineage's official documentation, listed in this lesson's Resources.

Exercises

Exercise 1 — Add the missing orders column that no place in the warehouse uses yet. Review LINEAGE_MAP: orders_s04's six columns (order_id, store_id, product_id, quantity, unit_price, order_ts) already all appear as the source of some fact_orders entry. Confirm this by counting how many times each orders column name appears within the dictionary's values.

from collections import Counter

all_sources = [source for sources in LINEAGE_MAP.values() for source in sources if source.startswith("orders.")]
print(Counter(all_sources))

Counter({'orders.quantity': 2, 'orders.unit_price': 2, 'orders.order_id': 1, 'orders.store_id': 1, 'orders.product_id': 1, 'orders.order_ts': 1})

Exercise 2 — Write a function that finds every derived column depending, even indirectly, on orders.unit_price. Using LINEAGE_MAP, write find_dependents(source_column, lineage_map) that returns the list of derived columns whose lineage includes source_column as one of their sources.

def find_dependents(source_column: str, lineage_map: dict[str, list[str]]) -> list[str]:
    return sorted([target for target, sources in lineage_map.items() if source_column in sources])

print(find_dependents("orders.unit_price", LINEAGE_MAP))

['fact_orders.revenue', 'fact_orders.unit_price']

Exercise 3 — Extend LINEAGE_MAP to explicitly distinguish between direct copy and transformation. Design a new structure — it could be a dictionary of tuples, or a second parallel dictionary — that captures, for every LINEAGE_MAP entry, whether the relationship is a direct copy or a transformation (like dim_store.country). No need to implement it fully; describe the structure and apply it to two or three example entries.

LINEAGE_MAP_WITH_TYPE = {
    "fact_orders.order_id": {"sources": ["orders.order_id"], "transformation": "direct_copy"},
    "fact_orders.revenue": {"sources": ["orders.quantity", "orders.unit_price"], "transformation": "quantity * unit_price"},
    "dim_store.country": {"sources": ["stores.city"], "transformation": "city_to_country_lookup"},
}

Summary and next step

In this lesson you built LINEAGE_MAP, this entire ecosystem's first complete lineage map: fifteen columns of Kiosko's warehouse, each traced back to its source column, including the distinction between direct copies and transformations (revenue, calculated from two sources; country, derived from city). You built trace_column() and lineage_report(), two ways of querying the same map — one specific, one complete — and you named OpenLineage and Marquez as this same work's production version, with its Dataset/Job/Run/facets model and Marquez as its reference implementation, with no server installed.

Before moving on you should be able to: explain LINEAGE_MAP's correct key-and-value direction (destination as key, source as value); name the warehouse's only column with more than one source, and why; and explain, in your own words, the difference between what LINEAGE_MAP does and what OpenLineage would automate in production.

Freshness, volume, and lineage — this module's three pieces — are already complete, each with executed evidence. Lesson 8's project brings them together into a single report, closing the module with the same honesty as earlier projects: what got covered, and exactly what this module never tried to be.

Resources

• OpenLineage — official documentation (the complete Dataset/Job/Run model and facets, the OpenAPI specification, and the Airflow/Spark/Flink/dbt integrations). openlineage.io/docs. In English.
• OpenLineage — official repository (GitHub, OpenLineage/OpenLineage) — source code and complete specification. github.com/OpenLineage/OpenLineage. In English.
• Marquez — project site (OpenLineage's reference implementation, originally open-sourced by WeWork, with its metadata server, visual interface, and lineage API). marquezproject.ai. In English.
• Module 1, lesson 5, of this same guide — the source of the deterministic city → country function, the transformation LINEAGE_MAP["dim_store.country"] traces. src/guides/data-reliability-and-governance-guide/workbook/module-01-when-green-does-not-mean-correct/en/05-meet-s04-kioskos-fourth-store.md. In English.
• Module 6, lesson 6, of this same guide — the complete conceptual argument this lesson turns into code. src/guides/data-reliability-and-governance-guide/workbook/module-06-freshness-volume-and-lineage/en/06-what-lineage-answers-that-a-contract-does-not.md. In English.
• This guide's DESIGN — LINEAGE_MAP's exact mandate and the boundary with OpenLineage/Marquez in production. src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.