Module 7: The Incident And Data Governance

Masking PII deterministically

Next →

Description

Lesson 5 deliberately left a question pending: ACCESS_POLICY decides whether a role can see a column, but once it can, that column still appears in plain text — the real email, the real phone number. For a role like finance or support, which really needs to contact a customer, that's correct. For a role like analyst, which only needs to correlate a customer's behavior over time — has this customer already bought before? — with no need to know who they really are, plain text is more access than its job requires. This lesson builds mask_pii(): a function that replaces every sensitive value with a deterministic hash — the same input value always produces the same result, using no source of randomness at all.

Connection to the module. Lesson 5 built control over which column each role sees. This lesson builds control over how a column looks once a role has permission to see it — the second of this module's three governance pieces, before lesson 7's minimal catalog.

An analogy: the customer's barcode, not their name on the label

Think of how a supermarket tracks a customer's repeat purchases in its loyalty program, with no need for the cashier to know who that person is. Every loyalty card has a code — a number or a barcode —, never the customer's name printed on any sales report. The system can perfectly answer "this code bought three times this month" or "code X and code Y bought the same product," with no analyst reviewing those reports ever needing to know whether code X corresponds to Ana or to Luis. And, importantly: a specific customer's code is always the same, every time they return to the store — if it changed every time, it would be impossible to track that it's the same person over time, and the whole loyalty program would stop making sense.

mask_pii() generates, precisely, that same kind of code: a value that replaces the sensitive data, stable over time — the same email always produces the same code —, useful for correlating without identifying. The difference with a physical barcode is that this code gets calculated mathematically from the original value, with a hash function — never assigned at random or stored in a separate table.

Worked example: mask_pii(), run twice to test determinism

# mask_pii.py -- module 7, lesson 6
import hashlib

import duckdb
import polars as pl

pl.Config.set_fmt_str_lengths(60)

MASK_SALT = "kiosko-mask-salt-2026"


def mask_pii(df: pl.DataFrame, columns: list[str], salt: str = MASK_SALT) -> pl.DataFrame:
    """Replaces every value in `columns` with a deterministic SHA-256 hash.

    Same value + same salt -> always the same hash. Never uses random or uuid4.
    """

    def _hash(value: str) -> str:
        return hashlib.sha256((salt + value).encode()).hexdigest()[:12]

    exprs = [pl.col(c).map_elements(_hash, return_dtype=pl.String).alias(c) for c in columns]
    return df.with_columns(exprs)


def main() -> None:
    con = duckdb.connect("kiosko.duckdb")
    customers_df = con.sql("SELECT * FROM customers").pl()

    print("=== customers, unmasked ===")
    print(customers_df)

    masked_once = mask_pii(customers_df, ["customer_email", "customer_phone"])
    print("\n=== mask_pii(customers_df, ['customer_email', 'customer_phone']) -- first run ===")
    print(masked_once)

    masked_twice = mask_pii(customers_df, ["customer_email", "customer_phone"])
    print("\n=== The same call, run a second time ===")
    print(masked_twice)

    print(f"\nBoth masked DataFrames are identical: {masked_once.equals(masked_twice)}")


if __name__ == "__main__":
    main()

What to expect (verified by actually running python3 mask_pii.py, with kiosko.duckdb containing customers, polars==1.43.2):

=== customers, unmasked ===
shape: (6, 3)
┌─────────────┬───────────────────┬───────────────────────────┐
│ customer_id ┆ customer_phone    ┆ customer_email            │
│ ---         ┆ ---                ┆ ---                       │
│ str         ┆ str                ┆ str                       │
╞═════════════╪═══════════════════╪═══════════════════════════╡
│ C001        ┆ +57-300-555-0101   ┆ ana.torres@example.com    │
│ C002        ┆ +51-999-555-0102   ┆ luis.rojas@example.com    │
│ C003        ┆ +56-9-5550-0103    ┆ maria.fuentes@example.com │
│ C004        ┆ +52-55-5550-0104   ┆ jorge.medina@example.com  │
│ C005        ┆ +57-300-555-0105   ┆ carla.suarez@example.com  │
│ C006        ┆ +51-999-555-0106   ┆ sofia.vargas@example.com  │
└─────────────┴───────────────────┴───────────────────────────┘

=== mask_pii(customers_df, ['customer_email', 'customer_phone']) -- first run ===
shape: (6, 3)
┌─────────────┬────────────────┬────────────────┐
│ customer_id ┆ customer_phone ┆ customer_email │
│ ---         ┆ ---            ┆ ---            │
│ str         ┆ str            ┆ str            │
╞═════════════╪════════════════╪════════════════╡
│ C001        ┆ 0a176df396c6   ┆ eb25122b17a2   │
│ C002        ┆ f3d770acd51b   ┆ aa87243c2e43   │
│ C003        ┆ 8e879c6fb8bc   ┆ 71b31bf92a73   │
│ C004        ┆ 083811ee9e8a   ┆ 93f1b0b8ab41   │
│ C005        ┆ f79999ac69ba   ┆ 3ae5fff0dc4f   │
│ C006        ┆ a7630ff3bdc8   ┆ b07fd99bf511   │
└─────────────┴────────────────┴────────────────┘

=== The same call, run a second time ===
shape: (6, 3)
┌─────────────┬────────────────┬────────────────┐
│ customer_id ┆ customer_phone ┆ customer_email │
│ ---         ┆ ---            ┆ ---            │
│ str         ┆ str            ┆ str            │
╞═════════════╪════════════════╪════════════════╡
│ C001        ┆ 0a176df396c6   ┆ eb25122b17a2   │
│ C002        ┆ f3d770acd51b   ┆ aa87243c2e43   │
│ C003        ┆ 8e879c6fb8bc   ┆ 71b31bf92a73   │
│ C004        ┆ 083811ee9e8a   ┆ 93f1b0b8ab41   │
│ C005        ┆ f79999ac69ba   ┆ 3ae5fff0dc4f   │
│ C006        ┆ a7630ff3bdc8   ┆ b07fd99bf511   │
└─────────────┴────────────────┴────────────────┘

Both masked DataFrames are identical: True

Read this result attentively, because it confirms mask_pii()'s central property with evidence, not just with prose: the two runs — separate, sharing no state at all beyond MASK_SALT, the same literal constant as always — produce exactly the same twelve masked values. ana.torres@example.com is always eb25122b17a2, in the first run, in the second, and in any future run of this same script, on any machine, as long as MASK_SALT doesn't change. This is what makes it possible for two different tables — or two runs of the same pipeline, on different days — to mask the same data consistently and remain joinable to each other by that masked value, with nobody needing to see the real email to confirm it's the same customer.

Diagram: the same value, always the same destination

flowchart LR
    A["ana.torres@example.com"] -->|"+ MASK_SALT"| B["sha256(...)"]
    B -->|".hexdigest()[:12]"| C["eb25122b17a2"]

    A2["ana.torres@example.com\n(another run, another day)"] -->|"+ MASK_SALT\n(the same constant)"| B2["sha256(...)"]
    B2 --> C2["eb25122b17a2\n(identical)"]

    D["luis.rojas@example.com"] -->|"+ MASK_SALT"| E["sha256(...)"]
    E --> F["aa87243c2e43\n(different -- different input value)"]

Going deeper: why this is pseudonymization, not anonymization — and why it matters

It's worth being precise with the vocabulary, because the distinction has real consequences. The European Union's General Data Protection Regulation (GDPR), in Article 4(5), defines pseudonymization as "the processing of personal data in such a manner that the personal data can no longer be attributed to a specific data subject without the use of additional information" — and clarifies that additional information must be kept separate and secured. mask_pii(), as written in this lesson, is exactly that: pseudonymization, not anonymization. The difference matters because pseudonymized data is still, technically, personal data — re-identifiable with the right additional information —, while genuinely anonymized data can never be linked back to the person under any circumstance.

What, in this case, is "the additional information" that would make reversing mask_pii()'s hash possible? MASK_SALT, combined with a dictionary of reasonable candidates. Run this experiment:

def mask_value(value: str, salt: str) -> str:
    return hashlib.sha256((salt + value).encode()).hexdigest()[:12]

email = "ana.torres@example.com"
print("With the correct MASK_SALT:  ", mask_value(email, "kiosko-mask-salt-2026"))
print("With a different salt:       ", mask_value(email, "otra-sal-cualquiera"))
With the correct MASK_SALT:   eb25122b17a2
With a different salt:        7eb3daf23ef9

Two completely different results, for the same email — it confirms whoever doesn't know MASK_SALT can't recalculate the correct hash even by accident. But notice the real problem: MASK_SALT is written, in this lesson and in Kiosko's production code as this guide built it, as a literal constant inside the source code. Anyone with access to Kiosko's repository — not just to the customers table — can read MASK_SALT directly, and with it, recalculate the hash of any email they want to try ("is customer C003 maria.fuentes@midominio.com? I'll try the hash and compare"). This is, precisely, what this guide's DESIGN warns from the start: mask_pii()'s goal is for the same data to always produce the same hash — pedagogical determinism, useful for making this guide reproducible byte for byte — not for it to be cryptographically irreversible against a real attacker. A real production system would keep MASK_SALT in a managed secret — outside the source code, with audited access and periodic rotation —, the same kind of control aws-core-services-guide / cloud-security-and-guardrails-guide teach for real credentials. This guide uses a literal constant, on purpose, because its goal is teaching the deterministic pseudonymization pattern, not building a production security system — the same honest distinction the DESIGN fixes from this guide's start.

Common mistakes

Not handling null values, and ending up "masking" the absence of data as if it were real data. What happens: someone writes a version of mask_pii() that calls str(value) with no check for whether value is None, and ends up generating a real hash for a row that actually has no email or phone. Confirm it with an experiment:

df_with_null = pl.DataFrame({"customer_id": ["C007"], "customer_email": [None]})

# correct version (map_elements with skip_nulls=True, the default)
correct_out = df_with_null.with_columns(
    pl.col("customer_email").map_elements(
        lambda v: hashlib.sha256((MASK_SALT + v).encode()).hexdigest()[:12], return_dtype=pl.String
    ).alias("customer_email")
)
print("skip_nulls=True (default):", correct_out["customer_email"].to_list())

# buggy version (skip_nulls=False, explicitly forced)
buggy_out = df_with_null.with_columns(
    pl.col("customer_email").map_elements(
        lambda v: hashlib.sha256((MASK_SALT + str(v)).encode()).hexdigest()[:12],
        return_dtype=pl.String, skip_nulls=False,
    ).alias("customer_email")
)
print("skip_nulls=False (bug):   ", buggy_out["customer_email"].to_list())
skip_nulls=True (default): [None]
skip_nulls=False (bug):    ['6ef8a9364ec5']

Why it happens: Polars's map_elements() already protects against this error by default (skip_nulls=True), so the bug only shows up if someone explicitly disables it, or writes the hash function by hand outside Polars with no check at all. How to spot it: 6ef8a9364ec5 is, visually, indistinguishable from any other real hash in this lesson — that's exactly the problem: a missing value got disguised as real data. How to fix it: mask_pii(), as written in this lesson's worked example, leaves Polars's default skip_nulls=True untouched — any null value in a PII column stays null after masking, never turning into a hash of the string "None".

Thinking a hash truncated to 12 characters ([:12]) is weaker or less "real" than the complete SHA-256 hash. What happens: someone notices SHA-256's complete hexdigest() has 64 characters, and that mask_pii() only uses the first 12, and concludes this makes masking significantly "less secure." Why it happens: more characters feels, intuitively, like more security. How to spot it: ask yourself how large the space of real values mask_pii() needs to distinguish actually is — Kiosko has six customers in this guide, and a real system of this size would have, at most, a few thousand. How to fix it: 12 hexadecimal characters represent 48 bits of information — more than 281 trillion possible combinations —, far more than enough to avoid accidental collisions (two different values with the same truncated hash) at any scale reasonable for Kiosko. Cutting to 12 characters is a readability decision — a 12-character hash fits in a printed table without breaking the format, as you already saw in this lesson's "What to expect" — not a security one. This scheme's real security depends, as Going deeper already explained, on how well protected MASK_SALT is, not on how many hash characters get kept.

Exercises

Exercise 1 — Run mask_pii.py yourself, and confirm determinism with a third run. In a new folder, with kiosko.duckdb containing customers, run python3 mask_pii.py twice in a row (two separate Python processes, not just two calls within the same script). Confirm ana.torres@example.com's hash is exactly eb25122b17a2 in both runs.

See solution

If MASK_SALT is still exactly "kiosko-mask-salt-2026", with no change at all, and customers has lesson 5's exact six rows, every run — no matter whether it's the same Python process or a completely new one, on the same machine or another — should produce exactly this lesson's same twelve hashes. This confirms something important the lesson didn't explicitly test: mask_pii()'s determinism doesn't depend on any internal Python state (like a dictionary's order or a random seed) — it depends solely on the two input values, the data and MASK_SALT, exactly how any pure function should behave.

Exercise 2 — Combine mask_pii() (this lesson) with ACCESS_POLICY (lesson 5) into a single build_role_view() function. The analyst role is allowed to see customer_email (according to ACCESS_POLICY), but has no legitimate reason to see it in plain text (according to this lesson's Going deeper section). Write build_role_view(df, role) that combines both pieces: filters columns with apply_access_policy(), and then masks customer_email/customer_phone if present, except for the "finance" and "support" roles, which do need the real value.

See solution 
PII_COLUMNS = {"customer_email", "customer_phone"}
ROLES_WITH_RAW_PII = {"finance", "support"}


def build_role_view(df: pl.DataFrame, role: str) -> pl.DataFrame:
    view = apply_access_policy(df, role)
    if role not in ROLES_WITH_RAW_PII:
        pii_present = [c for c in view.columns if c in PII_COLUMNS]
        if pii_present:
            view = mask_pii(view, pii_present)
    return view


print(build_role_view(customers_df, "analyst"))

Expected output:

shape: (6, 2)
┌─────────────┬────────────────┐
│ customer_id ┆ customer_email │
│ ---         ┆ ---            │
│ str         ┆ str            │
╞═════════════╪════════════════╡
│ C001        ┆ eb25122b17a2   │
│ C002        ┆ aa87243c2e43   │
│ C003        ┆ 71b31bf92a73   │
│ C004        ┆ 93f1b0b8ab41   │
│ C005        ┆ 3ae5fff0dc4f   │
│ C006        ┆ b07fd99bf511   │
└─────────────┴────────────────┘

analyst sees customer_email, exactly as ACCESS_POLICY allows — but masked, never the real email. If you call build_role_view(customers_df, "finance"), by contrast, you get all three columns with customer_email and customer_phone in plain text, with no hash at all — the exact combination of this module's two governance pieces: which column each role sees (lesson 5) and how it sees it (this lesson).

Exercise 3 — Argue whether MASK_SALT should be the same for customer_email and for customer_phone, or whether each column should have its own salt. In 2-3 sentences, considering this lesson's Going deeper vulnerability analysis (an attacker with MASK_SALT can try candidate values), argue whether using a single shared salt for both PII columns is a reasonable decision, or whether it introduces an additional risk a different salt per column would avoid.

See solution

Using the same MASK_SALT for both columns introduces no significant additional risk against an attacker who already knows the salt — if they can already recalculate hashes of candidate emails, recalculating hashes of candidate phone numbers with the same salt costs them no extra effort — so the security argument for separate salts is weak in this specific scheme. Where separate salts would matter is avoiding a different kind of problem: if, by coincidence, a real email and a real phone number had the same text-string value (an extremely unlikely case, but not impossible if someone reuses a free-text field for two purposes), a single shared salt would produce the same hash for both, which could confuse an analysis assuming hashes from different columns never coincide. In practice, for Kiosko's case — two columns with completely different data formats, email addresses and phone numbers, which could never coincide as text strings — a single shared MASK_SALT is a reasonable simplification, consistent with the rest of this guide, which prioritizes every piece being explainable with a single constant instead of managing a different salt per column.

Summary and next step

In this lesson you built mask_pii(), Kiosko's second data governance piece: a deterministic SHA-256 hash, with a fixed MASK_SALT, that replaces emails and phone numbers with a stable code — the same input value always produces the same result, confirmed by running the function twice on the same data. You learned the precise distinction between pseudonymization (what this lesson builds) and real anonymization (what GDPR requires for data to stop being considered personal), and why MASK_SALT as a literal constant in the source code is a deliberate pedagogical choice, not a production security recipe.

Before moving on you should be able to: explain, with this lesson's experiment, why changing MASK_SALT changes every resulting hash; cite GDPR Article 4(5)'s exact pseudonymization definition and explain why mask_pii() falls into that category, not anonymization; and combine ACCESS_POLICY (lesson 5) with mask_pii() (this lesson) into a single per-role view.

With this, Kiosko controls which column each role sees, and how it sees it. One last governance piece remains: documenting, in a single place, what tables exist, who owns each one, and which columns are sensitive — lesson 7 builds generate_catalog(), Kiosko's minimal catalog.

Resources

  • Python — official documentation, hashlib module (hashlib.sha256, .hexdigest(), mask_pii()'s complete technical foundation). docs.python.org/3/library/hashlib.html. In English.
  • European Union — General Data Protection Regulation (GDPR), Article 4(5) (the exact pseudonymization definition cited in this lesson's Going deeper section). gdpr-info.eu/art-4-gdpr. In English.
  • Polars — official documentation (map_elements, including the skip_nulls parameter that avoids this lesson's common mistake). docs.pola.rs. In English.
  • Module 5, lesson 1, of this same guide — the source of the "every rule explainable in a single sentence" principle this lesson applies to choosing a single shared MASK_SALT (Exercise 3). src/guides/data-reliability-and-governance-guide/workbook/module-05-accuracy-and-deterministic-anomaly-detection/en/01-module-introduction-5.md. In English.
  • This guide's DESIGN — the source of MASK_SALT and the explicit clarification that this guide's masking is pedagogical determinism, not production cryptography. src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.
← PreviousWho can see what: access control basicsNext →A minimal data catalog for Kiosko