A minimal data catalog for Kiosko

Description

After six modules and two governance lessons, Kiosko has a complete system of quality, contracts, lineage, quarantine, alerts, and access control — but all that information lives scattered across each module's code. If someone new at Kiosko asked today "what tables exist, who owns each one, and which have sensitive information?", they'd have to read eight complete modules to reconstruct the answer. This lesson closes that gap with this guide's third and last data governance piece: generate_catalog(), a function that gathers, into a single artifact (catalog.yaml), Kiosko's four tables, their owner, their columns, and which of those columns are PII.

Connection to the module. Lessons 5 and 6 built who can see which column, and how they see it. This lesson adds no new access control — it documents, in a single readable place, everything the earlier lessons already decided, so that information doesn't depend on someone remembering which module each rule got written in.

An analogy: the building directory, not the building itself

Think of a large office building's directory, the one at the entrance, with a list of every company, which floor it's on, and who its contact is. The directory isn't the building — it doesn't control who can enter each office, that's the keys' and access cards' job, already built in lessons 5 and 6 — it's a map that saves any new visitor from having to walk floor by floor, knocking on doors at random, to find what they're looking for. A data catalog fills exactly that role: it applies no control on its own, but it documents, in a centralized place, what exists and who's responsible for each thing — the difference between a governed system and a system where governance only lives in the memory of whoever built it.

Worked example: generate_catalog(), run over Kiosko's four tables

# generate_catalog.py -- module 7, lesson 7
import yaml

TABLES_SOURCE = [
    {
        "table": "orders_s04",
        "owner": "data-engineering@kiosko",
        "contract": "contracts/orders_contract.yaml",
        "columns": [
            {"name": "order_id", "sensitivity": "none"},
            {"name": "store_id", "sensitivity": "none"},
            {"name": "product_id", "sensitivity": "none"},
            {"name": "quantity", "sensitivity": "none"},
            {"name": "unit_price", "sensitivity": "none"},
            {"name": "order_ts", "sensitivity": "none"},
        ],
    },
    {
        "table": "dim_product",
        "owner": "data-engineering@kiosko",
        "contract": None,
        "columns": [
            {"name": "product_id", "sensitivity": "none"},
            {"name": "product_name", "sensitivity": "none"},
            {"name": "category", "sensitivity": "none"},
            {"name": "unit_cost", "sensitivity": "none"},
        ],
    },
    {
        "table": "dim_store",
        "owner": "data-engineering@kiosko",
        "contract": None,
        "columns": [
            {"name": "store_id", "sensitivity": "none"},
            {"name": "store_name", "sensitivity": "none"},
            {"name": "city", "sensitivity": "none"},
            {"name": "country", "sensitivity": "none"},
        ],
    },
    {
        "table": "customers",
        "owner": "delivery-app@kiosko",
        "contract": None,
        "columns": [
            {"name": "customer_id", "sensitivity": "none"},
            {"name": "customer_phone", "sensitivity": "pii"},
            {"name": "customer_email", "sensitivity": "pii"},
        ],
    },
]


def generate_catalog(tables: list[dict]) -> list[dict]:
    """Adds, to each table, how many of its columns are PII -- calculated, not hand-written."""
    catalog = []
    for t in tables:
        n_pii = sum(1 for c in t["columns"] if c["sensitivity"] == "pii")
        entry = dict(t)
        entry["n_columns"] = len(t["columns"])
        entry["n_pii_columns"] = n_pii
        catalog.append(entry)
    return catalog


def main() -> None:
    catalog = generate_catalog(TABLES_SOURCE)

    with open("catalog.yaml", "w") as f:
        yaml.safe_dump(catalog, f, sort_keys=False, allow_unicode=True)

    print("catalog.yaml written. Summary:")
    for entry in catalog:
        print(f"  {entry['table']:<12} owner={entry['owner']:<24} columns={entry['n_columns']} pii={entry['n_pii_columns']}")

    total_columns = sum(e["n_columns"] for e in catalog)
    total_pii = sum(e["n_pii_columns"] for e in catalog)
    print(f"\nTotal: {len(catalog)} tables, {total_columns} columns, {total_pii} PII columns")


if __name__ == "__main__":
    main()

What to expect (verified by actually running python3 generate_catalog.py, with pyyaml installed):

catalog.yaml written. Summary:
  orders_s04   owner=data-engineering@kiosko  columns=6 pii=0
  dim_product  owner=data-engineering@kiosko  columns=4 pii=0
  dim_store    owner=data-engineering@kiosko  columns=4 pii=0
  customers    owner=delivery-app@kiosko      columns=3 pii=2

Total: 4 tables, 17 columns, 2 PII columns

And the catalog.yaml file left written to disk, exactly:

- table: orders_s04
  owner: data-engineering@kiosko
  contract: contracts/orders_contract.yaml
  columns:
  - name: order_id
    sensitivity: none
  - name: store_id
    sensitivity: none
  - name: product_id
    sensitivity: none
  - name: quantity
    sensitivity: none
  - name: unit_price
    sensitivity: none
  - name: order_ts
    sensitivity: none
  n_columns: 6
  n_pii_columns: 0
- table: dim_product
  owner: data-engineering@kiosko
  contract: null
  columns:
  - name: product_id
    sensitivity: none
  - name: product_name
    sensitivity: none
  - name: category
    sensitivity: none
  - name: unit_cost
    sensitivity: none
  n_columns: 4
  n_pii_columns: 0
- table: dim_store
  owner: data-engineering@kiosko
  contract: null
  columns:
  - name: store_id
    sensitivity: none
  - name: store_name
    sensitivity: none
  - name: city
    sensitivity: none
  - name: country
    sensitivity: none
  n_columns: 4
  n_pii_columns: 0
- table: customers
  owner: delivery-app@kiosko
  contract: null
  columns:
  - name: customer_id
    sensitivity: none
  - name: customer_phone
    sensitivity: pii
  - name: customer_email
    sensitivity: pii
  n_columns: 3
  n_pii_columns: 2

Read this catalog carefully, because it answers, in a single file, three questions that until this lesson you could only answer by reading code: what tables exist (four, counting orders_s04 instead of the canonical week's generic orders, because it's this guide's incident's active table), who owns each one (data-engineering@kiosko for the warehouse's three tables; delivery-app@kiosko for customers, a different team — the same kind of responsibility separation any real organization has between systems), and which columns are sensitive (only two, out of seventeen total: customer_phone and customer_email, exactly the ones ACCESS_POLICY and mask_pii() have protected since lessons 5 and 6). Notice also the contract field: only orders_s04 has one, pointing to module 4's contracts/orders_contract.yaml — the other three tables still don't have a formal contract, an honest gap the catalog makes visible instead of hiding.

Diagram: the catalog as the map of everything else

flowchart TD
    subgraph Module4["Module 4"]
        A["contracts/orders_contract.yaml"]
    end
    subgraph Lesson5["Lesson 5"]
        B["ACCESS_POLICY"]
    end
    subgraph Lesson6["Lesson 6"]
        C["mask_pii() + MASK_SALT"]
    end

    A -.->|"referenced in"| D["catalog.yaml\n(this lesson)"]
    B -.->|"informs 'sensitivity: pii'"| D
    C -.->|"informs 'sensitivity: pii'"| D

    D --> E["orders_s04: has a contract,\n0 PII columns"]
    D --> F["dim_product, dim_store:\nno contract, 0 PII"]
    D --> G["customers: no contract,\n2 PII columns"]

Going deeper: this catalog's production version

generate_catalog() and catalog.yaml are, deliberately, the simplest possible version of an idea that in production has complete dedicated tools. The industry calls this category a metadata platform, and the best-known open-source example is DataHub, Apache-2.0 licensed, originating from LinkedIn's data infrastructure team and today maintained as an independent project. DataHub does, at a much larger scale and automatically, the same thing generate_catalog() does by hand in this lesson: it discovers tables by connecting directly to an organization's data systems (instead of someone hand-writing TABLES_SOURCE, as in this example), tracks lineage column by column — the same idea module 6's LINEAGE_MAP built manually —, and exposes a governance dashboard where you can flag which columns are sensitive and who owns each data asset.

This guide names DataHub, without installing it, for the exact same reason it already named Great Expectations and Soda in module 2, and OpenLineage/Marquez in module 6: keeping the complete ecosystem at $0, with no account or server, while making clear what the production path is when Kiosko — or any real organization — grows beyond what a hand-written Python dictionary can sustain. The difference between generate_catalog() and DataHub isn't conceptual — both answer "what exists, who owns it, what's sensitive" — it's one of scale and automation: DataHub connects to real systems and keeps itself updated; TABLES_SOURCE, in this lesson, has to be updated by hand every time Kiosko adds a new column — a real, honest, and acceptable limitation for this case study's size.

Common mistakes

Thinking catalog.yaml applies some control on its own, as if it were a second copy of ACCESS_POLICY. What happens: someone, seeing sensitivity: pii in the catalog, expects that fact alone to be enough for customer_email to end up protected — with no need for ACCESS_POLICY or mask_pii() to reference it too. Why it happens: the catalog feels, superficially, like a "central" source of truth that should be enough on its own. How to spot it: check whether any code in this guide reads catalog.yaml to decide, at runtime, whether to mask a column — it doesn't; ACCESS_POLICY and PII_COLUMNS (lesson 6) are completely independent dictionaries, written separately. How to fix it: catalog.yaml is structured documentation, not an enforcement mechanism — it describes what should be true, the same way on_violation: quarantine in module 4's contract declared an intention without executing it on its own. A real production system (like DataHub) can indeed connect the catalog to real access control, but that integration falls outside this guide's scope.

Writing TABLES_SOURCE with customers's columns in the same order they appear in DuckDB's real table, and assuming that matters for generate_catalog(). What happens: someone manually reorders TABLES_SOURCE's columns to exactly match CREATE TABLE customers (...)'s order, thinking a mismatch would break something. Why it happens: in many other contexts in this guide — like Pandera's schema — column order can indeed matter. How to spot it: check what generate_catalog() does with each table's columns list — it only counts how many are "pii" and how many total, never comparing their order against any external source. How to fix it: TABLES_SOURCE, in this lesson, is a manual schema description, completely disconnected from the real table in kiosko.duckdb — its order doesn't affect generate_catalog()'s result at all. This disconnect is, precisely, the limitation this lesson's Going deeper section already named as the difference from a real production catalog: nothing in this example automatically verifies TABLES_SOURCE still reflects each table's true structure.

Exercises

Exercise 1 — Run generate_catalog.py yourself, from scratch. In a new folder, with pyyaml installed (pip install pyyaml), run python3 generate_catalog.py. Confirm the printed summary matches this lesson's exactly, and open the resulting catalog.yaml to confirm it has four entries.

Exercise 2 — Add a tables_missing_contract(catalog) -> list[str] function that reports, calculated instead of hand-written, which catalog tables still don't have a formal contract. Using generate_catalog()'s result, filter tables whose contract field is None.

def tables_missing_contract(catalog: list[dict]) -> list[str]:
    return [entry["table"] for entry in catalog if entry["contract"] is None]

missing = tables_missing_contract(catalog)
print(f"Tables with no formal contract: {missing}")

Tables with no formal contract: ['dim_product', 'dim_store', 'customers']

Exercise 3 — Argue whether generate_catalog() should include, besides sensitivity: pii or sensitivity: none, a third intermediate level like sensitivity: internal (columns that don't identify a person, but that Kiosko also doesn't want to make public, like dim_product's unit_cost). In 2-3 sentences, argue for or against adding that third level, considering the "every rule explainable in one sentence" principle that has guided the rest of this guide.

Summary and next step

In this lesson you built generate_catalog(), this guide's third and last data governance piece: a minimal catalog that gathers, into a single catalog.yaml, Kiosko's four tables, their owner, their columns, and which are sensitive — calculated, not hand-written, from an explicit description of each table's schema. You named DataHub as this same idea's production version, with the same "name it, don't install it" discipline Great Expectations, Soda, OpenLineage, and Marquez already practiced in earlier modules.

Before moving on you should be able to: explain why catalog.yaml is documentation, not an enforcement mechanism, citing what generate_catalog() does and doesn't do; reproduce, by running the code yourself, the complete four-table catalog; and calculate, with code, which Kiosko tables still don't have a formal contract.

With this, this module's three governance pieces are complete: ACCESS_POLICY decides which column each role sees (lesson 5), mask_pii() decides how a sensitive column looks (lesson 6), and generate_catalog() documents everything in one place (this lesson). And the incident's three pieces are complete too: quarantine() separates good from bad (lesson 3), raise_alert() and runbook.md say what to do about it (lesson 4). The closing project, lesson 8, brings all six pieces together into a single incident-response script, run end to end against S04's real incident.

Resources

• PyYAML — official documentation (yaml.safe_dump, used to write catalog.yaml). pyyaml.org/wiki/PyYAMLDocumentation. In English.
• DataHub — official repository (GitHub, datahub-project/datahub) (Apache-2.0 license, the open-source metadata catalog named in this lesson's Going deeper section as generate_catalog()'s production version). github.com/datahub-project/datahub. In English.
• Module 4, lesson 3, of this same guide — the source of contracts/orders_contract.yaml, referenced in this lesson's contract field. src/guides/data-reliability-and-governance-guide/workbook/module-04-data-contracts-as-versioned-artifacts/en/03-writing-orders-contract-yaml.md. In English.
• This guide's DESIGN — the source of generate_catalog() and the mandate for a minimal table/owner/columns/sensitivity/contract catalog. src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.