Module 8: Project Kioskos Reliability And Governance System

Publishing the catalog and access policy

Description

Lessons 4 and 5 ran run_full_gate() in both directions — this module's technical core is already complete. This lesson publishes the other half of Kiosko's trust system, the one that does not depend on which file arrived today: the data governance module 7 built — lineage, catalog, role-based access, PII masking — runs exactly the same over S04, over the clean day, or over any file Kiosko receives next year. A trust system isn't just one that detects problems in the data coming in — it's also one that documents what tables exist, who owns each one, and who can see which column, no matter whether that table had an incident yesterday or never had any.

Connection to the module. Lessons 3 through 5 answered "is this data correct?". This lesson answers the two questions module 7 already raised and no run_full_gate() result answers on its own: "where does every column come from?" and "who can see it?". Lesson 7 closes the module with what remains pending even after answering all three questions together.

An analogy: the building directory, not just the fire alarms

This module's lessons 3 through 5 built a building's fire alarm system: it detects smoke, separates people at risk, notifies the fire department. But a safe building needs more than alarms — it needs a directory: a map of what's on each floor, who owns each office, and a list of who has the key to each door. That directory reacts to no fire — it always exists, regardless of whether there's an emergency today or not. This lesson builds that directory for Kiosko: catalog.yaml documents which tables exist and who owns each one; ACCESS_POLICY and mask_pii() document and apply who has the key to each column.

Worked example: lineage, catalog, access, and masking, published together

Step 1 — `LINEAGE_MAP`, with no change at all since module 6

# add to kiosko_trust.py
LINEAGE_MAP = {
    "fact_orders.order_id": ["orders.order_id"],
    "fact_orders.store_id": ["orders.store_id"],
    "fact_orders.product_id": ["orders.product_id"],
    "fact_orders.quantity": ["orders.quantity"],
    "fact_orders.unit_price": ["orders.unit_price"],
    "fact_orders.revenue": ["orders.quantity", "orders.unit_price"],
    "fact_orders.order_ts": ["orders.order_ts"],
    "dim_product.product_id": ["products.product_id"],
    "dim_product.product_name": ["products.product_name"],
    "dim_product.category": ["products.category"],
    "dim_product.unit_cost": ["products.unit_cost"],
    "dim_store.store_id": ["stores.store_id"],
    "dim_store.store_name": ["stores.store_name"],
    "dim_store.city": ["stores.city"],
    "dim_store.country": ["stores.city"],
}

LINEAGE_MAP doesn't change between S04 and the clean day, or between any future file — it describes how the warehouse's columns get built from source columns, a relationship living at the schema level, not at any specific gate run. It's, precisely, the same reason lineage gets documented once, in a single place, instead of recalculated every time a file arrives.

Step 2 — `ACCESS_POLICY` and `mask_pii()`, with no change at all since module 7

ACCESS_POLICY: dict[str, list[str]] = {
    "analyst": ["order_id", "store_id", "product_id", "quantity", "unit_price", "order_ts", "customer_id", "customer_email"],
    "finance": ["order_id", "store_id", "product_id", "quantity", "unit_price", "order_ts", "customer_id", "customer_email", "customer_phone"],
    "support": ["order_id", "customer_id", "customer_email", "customer_phone"],
}
PII_COLUMNS = {"customer_email", "customer_phone"}
ROLES_WITH_RAW_PII = {"finance", "support"}


def apply_access_policy(df: pl.DataFrame, role: str, policy: dict = ACCESS_POLICY) -> pl.DataFrame:
    allowed = [c for c in policy[role] if c in df.columns]
    return df.select(allowed)


def mask_pii(df: pl.DataFrame, columns: list[str], salt: str = MASK_SALT) -> pl.DataFrame:
    def _hash(value: str) -> str:
        return hashlib.sha256((salt + value).encode()).hexdigest()[:12]
    exprs = [pl.col(c).map_elements(_hash, return_dtype=pl.String).alias(c) for c in columns]
    return df.with_columns(exprs)


def build_role_view(df: pl.DataFrame, role: str) -> pl.DataFrame:
    view = apply_access_policy(df, role)
    if role not in ROLES_WITH_RAW_PII:
        pii_present = [c for c in view.columns if c in PII_COLUMNS]
        if pii_present:
            view = mask_pii(view, pii_present)
    return view

Step 3 — `generate_catalog()`, now linking each table to its contract

def generate_catalog(tables: list[dict]) -> list[dict]:
    catalog = []
    for t in tables:
        n_pii = sum(1 for c in t["columns"] if c["sensitivity"] == "pii")
        entry = dict(t)
        entry["n_columns"] = len(t["columns"])
        entry["n_pii_columns"] = n_pii
        catalog.append(entry)
    return catalog


TABLES_SOURCE = [
    {
        "table": "orders_s04", "owner": "data-engineering@kiosko", "contract": "contracts/orders_contract.yaml",
        "columns": [
            {"name": "order_id", "sensitivity": "none"}, {"name": "store_id", "sensitivity": "none"},
            {"name": "product_id", "sensitivity": "none"}, {"name": "quantity", "sensitivity": "none"},
            {"name": "unit_price", "sensitivity": "none"}, {"name": "order_ts", "sensitivity": "none"},
        ],
    },
    {
        "table": "dim_product", "owner": "data-engineering@kiosko", "contract": None,
        "columns": [
            {"name": "product_id", "sensitivity": "none"}, {"name": "product_name", "sensitivity": "none"},
            {"name": "category", "sensitivity": "none"}, {"name": "unit_cost", "sensitivity": "none"},
        ],
    },
    {
        "table": "dim_store", "owner": "data-engineering@kiosko", "contract": None,
        "columns": [
            {"name": "store_id", "sensitivity": "none"}, {"name": "store_name", "sensitivity": "none"},
            {"name": "city", "sensitivity": "none"}, {"name": "country", "sensitivity": "none"},
        ],
    },
    {
        "table": "customers", "owner": "delivery-app@kiosko", "contract": None,
        "columns": [
            {"name": "customer_id", "sensitivity": "none"}, {"name": "customer_phone", "sensitivity": "pii"},
            {"name": "customer_email", "sensitivity": "pii"},
        ],
    },
]

Notice the one detail this lesson adds compared to module 7: TABLES_SOURCE's orders_s04 entry declares "contract": "contracts/orders_contract.yaml", not None. This connects, for the first time inside the catalog itself, the table run_full_gate() validates with the artifact declaring what's expected of it — anyone consulting catalog.yaml immediately knows orders_s04 has a formal contract behind it, while dim_product, dim_store, and customers still don't (a real improvement opportunity, not a bug — module 4 only built a contract for orders, on purpose, leaving the rest outside this guide's scope).

Step 4 — the complete run

# publish_governance.py
import duckdb
import polars as pl
import yaml

from kiosko_trust import ACCESS_POLICY, LINEAGE_MAP, build_role_view, generate_catalog

pl.Config.set_fmt_str_lengths(60)

con = duckdb.connect("kiosko.duckdb")
customers_df = con.sql("SELECT * FROM customers").pl()

print("=== 1. LINEAGE_MAP -- complete report ===")
rows = [
    {"derived_column": target, "source_columns": ", ".join(sources), "n_sources": len(sources)}
    for target, sources in LINEAGE_MAP.items()
]
lineage_df = pl.DataFrame(rows).sort("derived_column")
pl.Config.set_tbl_rows(20)
print(lineage_df)
print(f"\nTotal mapped columns: {len(LINEAGE_MAP)}")

print("\n=== 2. ACCESS_POLICY + mask_pii() -- one view per role ===")
for role in ["analyst", "finance", "support"]:
    view = build_role_view(customers_df, role)
    print(f"  role={role:<8} columns={view.columns}")

print(f"\n  Detail, role=analyst (masked email):")
print(build_role_view(customers_df, "analyst"))

print("\n=== 3. generate_catalog() -- catalog.yaml ===")
catalog = generate_catalog(TABLES_SOURCE)
with open("catalog.yaml", "w") as f:
    yaml.safe_dump(catalog, f, sort_keys=False, allow_unicode=True)
total_pii = sum(e["n_pii_columns"] for e in catalog)
print(f"catalog.yaml written: {len(catalog)} tables, {sum(e['n_columns'] for e in catalog)} columns, {total_pii} PII columns")
for entry in catalog:
    print(f"  {entry['table']:<12} owner={entry['owner']:<26} contract={entry['contract']}")

What to expect (verified by actually running python3 publish_governance.py, with kiosko.duckdb containing customers, pyyaml installed):

=== 1. LINEAGE_MAP -- complete report ===
shape: (15, 3)
┌──────────────────────────┬────────────────────────────────────┬───────────┐
│ derived_column           ┆ source_columns                     ┆ n_sources │
│ ---                      ┆ ---                                ┆ ---       │
│ str                      ┆ str                                ┆ i64       │
╞══════════════════════════╪════════════════════════════════════╪═══════════╡
│ dim_product.category     ┆ products.category                  ┆ 1         │
│ dim_product.product_id   ┆ products.product_id                ┆ 1         │
│ dim_product.product_name ┆ products.product_name              ┆ 1         │
│ dim_product.unit_cost    ┆ products.unit_cost                 ┆ 1         │
│ dim_store.city           ┆ stores.city                        ┆ 1         │
│ dim_store.country        ┆ stores.city                        ┆ 1         │
│ dim_store.store_id       ┆ stores.store_id                    ┆ 1         │
│ dim_store.store_name     ┆ stores.store_name                  ┆ 1         │
│ fact_orders.order_id     ┆ orders.order_id                    ┆ 1         │
│ fact_orders.order_ts     ┆ orders.order_ts                    ┆ 1         │
│ fact_orders.product_id   ┆ orders.product_id                  ┆ 1         │
│ fact_orders.quantity     ┆ orders.quantity                    ┆ 1         │
│ fact_orders.revenue      ┆ orders.quantity, orders.unit_price ┆ 2         │
│ fact_orders.store_id     ┆ orders.store_id                    ┆ 1         │
│ fact_orders.unit_price   ┆ orders.unit_price                  ┆ 1         │
└──────────────────────────┴────────────────────────────────────┴───────────┘

Total mapped columns: 15

=== 2. ACCESS_POLICY + mask_pii() -- one view per role ===
  role=analyst  columns=['customer_id', 'customer_email']
  role=finance  columns=['customer_id', 'customer_email', 'customer_phone']
  role=support  columns=['customer_id', 'customer_email', 'customer_phone']

  Detail, role=analyst (masked email):
shape: (6, 2)
┌─────────────┬────────────────┐
│ customer_id ┆ customer_email │
│ ---         ┆ ---            │
│ str         ┆ str            │
╞═════════════╪════════════════╡
│ C001        ┆ eb25122b17a2   │
│ C002        ┆ aa87243c2e43   │
│ C003        ┆ 71b31bf92a73   │
│ C004        ┆ 93f1b0b8ab41   │
│ C005        ┆ 3ae5fff0dc4f   │
│ C006        ┆ b07fd99bf511   │
└─────────────┴────────────────┘

=== 3. generate_catalog() -- catalog.yaml ===
catalog.yaml written: 4 tables, 17 columns, 2 PII columns
  orders_s04   owner=data-engineering@kiosko     contract=contracts/orders_contract.yaml
  dim_product  owner=data-engineering@kiosko     contract=None
  dim_store    owner=data-engineering@kiosko     contract=None
  customers    owner=delivery-app@kiosko         contract=None

Notice something worth confirming carefully: none of these three blocks — lineage, access, catalog — made any reference to S04, to the clean day, or to any run_full_gate() result. They're, quite deliberately, independent of which file arrived today. That independence is, precisely, what turns them into governance instead of detection: they describe Kiosko's structure and access rules permanently, not a specific run's state.

Diagram: two independent layers, one system

flowchart TB
    subgraph Detection["Detection layer (lessons 3-5)"]
        direction LR
        D1["orders_2026-08-14.csv"] --> G["run_full_gate()"]
        D2["orders_2026-08-15.csv"] --> G
        G --> R["6 failures / 0 failures"]
    end

    subgraph Governance["Governance layer (this lesson)"]
        direction LR
        L["LINEAGE_MAP"]
        A["ACCESS_POLICY + mask_pii()"]
        C["generate_catalog()"]
    end

    Detection -.->|"does NOT depend on this"| Governance
    Governance -.->|"runs the same, regardless\nof the gate's result"| Detection

Common mistakes

Expecting catalog.yaml to include run_full_gate()'s result, as if it were a data quality report. What happens: someone, seeing catalog.yaml mention contracts/orders_contract.yaml for orders_s04, expects to also find the last run's failure count, or the quarantine status. Why it happens: lessons 4 and 5 just showed concrete numeric results (6, 0), and it's natural to expect the catalog to gather them too. How to spot it: review generate_catalog()'s structure — table, owner, contract, columns, n_columns, n_pii_columns — no field describes any specific gate run's result. How to fix it: the catalog documents what exists (tables, owners, contracts, column sensitivity), not what happened in a particular run — that's a different kind of information, which in a real production system would live in a separate execution history (the territory of airflow-and-declarative-orchestration-guide, named in lesson 7).

Applying mask_pii() to orders_s04 "just in case it has some sensitive data." What happens: someone, generalizing customers's pattern, calls mask_pii(orders_df, ["order_id"]) or similar, thinking masking more columns is always safer. Why it happens: after seeing mask_pii() work well on customer_email/customer_phone, it seems reasonable to apply it more broadly. How to spot it: review this lesson's TABLES_SOURCE — no column in orders_s04, dim_product, or dim_store has sensitivity: "pii"; only customer_phone and customer_email in customers. How to fix it: masking a column that identifies no real person adds no protection — it just makes order_id (a transaction identifier, not a person's) unreadable with no privacy benefit at all. mask_pii() applies exclusively to columns the catalog already classified as pii, never "just in case."

Exercises

Exercise 1 — Run publish_governance.py yourself, from scratch. With kiosko.duckdb (containing customers) in your folder, run this lesson's complete script. Confirm you see exactly 15 mapped lineage columns and 4 tables in the catalog, with 2 PII columns.

See solution

If customers has module 7, lesson 5's exact six rows, the output should reproduce this lesson's exactly: LINEAGE_MAP's fifteen rows, the three role views with the correct columns (analyst with the masked email, finance and support in plain text), and catalog.yaml with 4 tables, 17 total columns, 2 PII. If your result differs in mask_pii()'s hashes, first check MASK_SALT is still exactly "kiosko-mask-salt-2026" — any change there alters every hash with no other number changing.

Exercise 2 — Add a contract for dim_product to the catalog, and decide what fields it would need. dim_product today has "contract": None in TABLES_SOURCE. Without writing the complete YAML, describe in 2-3 sentences what rules would make sense to declare for that table, based on what you already know from orders_contract.yaml (module 4).

See solution

A reasonable contract for dim_product would declare, at minimum: product_id unique and non-null (the key validate_referential_integrity() already uses for the anti-join against orders); unit_cost non-negative (minimum: 0, the same pattern as unit_price in orders_contract.yaml); and possibly category restricted to a closed list of known values (beverages, snacks, electronics), something this guide's current ColumnContract doesn't support yet — it would be a reasonable extension, not part of this guide's scope. dim_product's SLA would be different in nature from orders_s04's: it makes no sense to talk about "24-hour freshness" for a catalog table that changes much less frequently than daily orders; a more appropriate SLA might measure, instead, how much time can pass without someone confirming the catalog is still current.

Exercise 3 — Explain why LINEAGE_MAP, ACCESS_POLICY, and generate_catalog() do NOT receive df (the orders DataFrame) as a parameter, unlike the seven functions run_full_gate() orchestrates. In 2-3 sentences, based on this lesson's "Two independent layers" diagram, explain the design difference.

See solution

The seven functions run_full_gate() orchestrates answer questions about concrete data that arrived today — they need df because their result depends, row by row, on what that specific file contains. LINEAGE_MAP, ACCESS_POLICY, and the TABLES_SOURCE definition feeding generate_catalog() answer questions about Kiosko's permanent structure — where each column comes from, who can see each one, what tables exist —, information that doesn't change from one run to the next, no matter whether today's file had six problems or none. That's why these three pieces get defined as constants or applied over reference tables (customers), never over run_full_gate()'s result — mixing the two layers would turn permanent information into something that would change, with no reason, every time a new file arrives.

Summary and next step

In this lesson you published Kiosko's complete system's governance layer: LINEAGE_MAP with the warehouse's fifteen columns traced to their source, ACCESS_POLICY and mask_pii() applied over customers with three distinct role views, and generate_catalog() writing catalog.yaml with Kiosko's four tables — now, for the first time, linking orders_s04 with its formal contract. You confirmed, with evidence, this layer is completely independent from run_full_gate()'s result: it applies the same over S04, over the clean day, or over any file arriving afterward.

Before moving on you should be able to: explain why the catalog includes no gate run's result; and name customers's three distinct role views (analyst, finance, support) and which columns each one sees.

Lesson 7 closes the module with the honest map of what Kiosko still needs, even with the seven checks, quarantine, alert, and complete governance already working: which sibling guides in this ecosystem solve each boundary this guide, by design, decided not to cross.

Resources

Module 6, lesson 7, of this same guide — the exact source of LINEAGE_MAP. src/guides/data-reliability-and-governance-guide/workbook/module-06-freshness-volume-and-lineage/en/07-mapping-kioskos-lineage-by-hand.md. In English.
Module 7, lessons 5-7, of this same guide — the exact source of ACCESS_POLICY, mask_pii(), and generate_catalog(). src/guides/data-reliability-and-governance-guide/workbook/module-07-the-incident-and-data-governance/en/. In English.
PyYAML — official documentation (yaml.safe_dump, used to write catalog.yaml). pyyaml.org/wiki/PyYAMLDocumentation. In English.
This guide's DESIGN. src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.

← PreviousRunning the full gate against a clean day Next →What Kiosko still needs

Publishing the catalog and access policy

Description

An analogy: the building directory, not just the fire alarms

Worked example: lineage, catalog, access, and masking, published together

Step 1 — LINEAGE_MAP, with no change at all since module 6

Step 2 — ACCESS_POLICY and mask_pii(), with no change at all since module 7

Step 3 — generate_catalog(), now linking each table to its contract

Step 4 — the complete run

Diagram: two independent layers, one system

Common mistakes

Exercises

Summary and next step

Resources

Step 1 — `LINEAGE_MAP`, with no change at all since module 6

Step 2 — `ACCESS_POLICY` and `mask_pii()`, with no change at all since module 7

Step 3 — `generate_catalog()`, now linking each table to its contract