Module 1: When Green Does Not Mean Correct

What foundations' `validate_orders()` already caught

Description

data-engineering-foundations-guide (module 5) built validate_orders(): four combined checks — check_schema(), check_nulls_and_types(), check_business_rules(), and duplicate order_id detection — that split a list of raw rows into valid and rejected, without any single broken row stopping the processing of the rest. This lesson doesn't rewrite a single line of that function. It brings it in exactly as it is, and classifies it with the vocabulary you just built in lesson 3: of the six dimensions of data quality, which ones does validate_orders() actually answer, and which did it never promise to answer?

Connection to the module. This is the bridge lesson between the vocabulary (lessons 2-3) and the real diagnosis (lessons 5-8). Without this precise classification, running validate_orders() against S04's file in lesson 6 would just be "seeing what happens" — with this lesson, you already know, ahead of time, what you should expect it to catch and what you should expect to slip through, and why.

An analogy: the front-door guard, not the full auditor

An office building has a guard at the main entrance. Their job is concrete and limited: confirm each person has a visible badge, that the badge isn't expired, and that the name on the badge matches — at a glance — the person carrying it. That guard does their job well if they cover exactly those three things. What that guard doesn't do — and never promised to do — is verify whether the person, once inside, is authorized to enter the server room, or whether the document in their backpack is the right one, or whether the meeting they say they're going to actually exists on someone's calendar. Those questions need additional controls, at other points in the building, with other tools.

validate_orders() is that front-door guard. It asks exactly four questions, well-executed, about every row trying to enter Kiosko's pipeline: does it have all the fields? Do those fields have the right content and type? Do the values respect basic business rules? Has this identifier already come in before? None of those four questions is "does this row's product_id actually exist in the product catalog?" or "does this price make sense compared to what this product normally costs?" — those questions need a different control, at a different point in the system. It's not that the front-door guard is failing at their job. It's that their job, by design, never included those questions.

Worked example: rebuilding `validate_orders()`, already familiar

This is exactly the code you already built, piece by piece, in data-engineering-foundations-guide (module 5, lessons 3 through 6) — copy it exactly as it is, with no changes, into a new kiosko.py file for this guide:

# kiosko.py -- exactly the same code from data-engineering-foundations-guide, module 5
from dataclasses import dataclass
from datetime import datetime

STORES = [
    {"store_id": "S01", "store_name": "Kiosko Centro", "city": "Bogota"},
    {"store_id": "S02", "store_name": "Kiosko Norte", "city": "Lima"},
    {"store_id": "S03", "store_name": "Kiosko Sur", "city": "Santiago"},
]

PRODUCTS = [
    {"product_id": "P001", "product_name": "Bottled Water 600ml", "category": "beverages", "unit_cost": 0.40},
    {"product_id": "P002", "product_name": "Energy Bar", "category": "snacks", "unit_cost": 0.60},
    {"product_id": "P003", "product_name": "Instant Coffee Sachet", "category": "beverages", "unit_cost": 0.35},
    {"product_id": "P004", "product_name": "Phone Charger Cable", "category": "electronics", "unit_cost": 2.10},
]


@dataclass
class Order:
    order_id: str
    store_id: str
    product_id: str
    quantity: int
    unit_price: float
    order_ts: datetime


def parse_order(row: dict) -> Order:
    return Order(
        order_id=row["order_id"],
        store_id=row["store_id"],
        product_id=row["product_id"],
        quantity=int(row["quantity"]),
        unit_price=float(row["unit_price"]),
        order_ts=datetime.fromisoformat(row["order_ts"]),
    )


REQUIRED_FIELDS = ["order_id", "store_id", "product_id", "quantity", "unit_price", "order_ts"]


def check_schema(row: dict) -> list[str]:
    reasons = []
    for field in REQUIRED_FIELDS:
        if field not in row:
            reasons.append(f"missing required field '{field}'")
    return reasons


def check_nulls_and_types(row: dict) -> list[str]:
    reasons = []
    for field in REQUIRED_FIELDS:
        if row[field] == "" or row[field] is None:
            reasons.append(f"field '{field}' is null or empty")
    if reasons:
        return reasons

    try:
        int(row["quantity"])
    except ValueError:
        reasons.append(f"field 'quantity' is not a valid integer: '{row['quantity']}'")
    try:
        float(row["unit_price"])
    except ValueError:
        reasons.append(f"field 'unit_price' is not a valid number: '{row['unit_price']}'")
    try:
        datetime.fromisoformat(row["order_ts"])
    except ValueError:
        reasons.append(f"field 'order_ts' is not a valid ISO 8601 timestamp: '{row['order_ts']}'")
    return reasons


def check_business_rules(order: Order) -> list[str]:
    reasons = []
    if order.quantity <= 0:
        reasons.append(f"quantity must be > 0, got {order.quantity}")
    if order.unit_price < 0:
        reasons.append(f"unit_price must be >= 0, got {order.unit_price}")
    return reasons


def validate_orders(rows: list[dict]) -> tuple[list[dict], list[dict]]:
    valid: list[dict] = []
    rejected: list[dict] = []
    seen_order_ids: set[str] = set()

    for row in rows:
        reasons: list[str] = []

        reasons += check_schema(row)
        if reasons:
            rejected.append({"row": row, "reasons": reasons})
            continue

        reasons += check_nulls_and_types(row)
        if reasons:
            rejected.append({"row": row, "reasons": reasons})
            continue

        order = parse_order(row)
        reasons += check_business_rules(order)

        if order.order_id in seen_order_ids:
            reasons.append(f"duplicate order_id '{order.order_id}'")

        if reasons:
            rejected.append({"row": row, "reasons": reasons})
            continue

        seen_order_ids.add(order.order_id)
        valid.append(row)

    return valid, rejected


def print_validation_report(valid: list[dict], rejected: list[dict]) -> None:
    total = len(valid) + len(rejected)
    print("=== Kiosko: validation report ===")
    print(f"Total rows: {total}")
    print(f"Valid: {len(valid)}")
    print(f"Rejected: {len(rejected)}\n")
    if rejected:
        print("=== Rejected row details ===")
        for item in rejected:
            order_id = item["row"].get("order_id", "???")
            print(f"{order_id}:")
            for reason in item["reasons"]:
                print(f"  - {reason}")

Notice this block has no "What to expect" — it doesn't need one yet: it's exactly the same code you already ran and verified in data-engineering-foundations-guide, quoted verbatim, with no changes. Its behavior on known data is already confirmed in that guide. What's new in this lesson isn't the code — it's the table that follows.

The exact classification: what each piece asks

Function	What it checks	Dimension(s) it covers
`check_schema()`	Do the six required keys exist in the row's dictionary?	Completeness (partial: only key presence)
`check_nulls_and_types()`	Are the values non-empty, and do they convert to the correct type?	Completeness (empty content) + Validity (type)
`check_business_rules()`	Is `quantity > 0`? Is `unit_price >= 0`?	Validity (business range)
`order_id` check against `seen_order_ids`	Has this identifier already been seen in the same batch?	Uniqueness

Four functions, and between them they cover exactly three of the six dimensions lesson 3 defined: completeness, validity, and uniqueness. None of the four asks anything about consistency — whether product_id actually exists in PRODUCTS —, about accuracy — whether unit_price makes sense compared to what that product normally costs —, or about freshness — whether the file arrived within the expected time window. It isn't an oversight: go back to lesson 5 of foundations' module 5 (the one that built check_business_rules()) and you'll find, in its own exercise 3, the question "how would you catch a product_id that doesn't exist?" posed explicitly without building the answer yet — foundations already knew that question was left open.

Diagram: the four pieces against the six dimensions

flowchart LR
    subgraph CUBIERTO["What validate_orders() DOES answer"]
        A["check_schema()"] --> D1["Completeness\n(partial)"]
        B["check_nulls_and_types()"] --> D1
        B --> D2["Validity\n(type)"]
        C["check_business_rules()"] --> D2b["Validity\n(business range)"]
        E["seen_order_ids"] --> D3["Uniqueness"]
    end

    subgraph SIN_CUBRIR["What validate_orders() NEVER promised to answer"]
        D4["Consistency\n(reference to another table)"]
        D5["Accuracy\n(vs. a baseline)"]
        D6["Freshness\n(vs. a time window)"]
    end

The diagram, deliberately, draws no arrow from the four functions to the block on the right — because there is none. That is, precisely, this lesson's central finding: it's not that validate_orders() tries to answer consistency, accuracy, or freshness and fails — it's that those three questions are completely outside its design, from the very first day it was written, in an earlier guide of this ecosystem.

Going deeper: why `check_business_rules()` doesn't check `product_id`

Go back to check_business_rules()'s exact code:

def check_business_rules(order: Order) -> list[str]:
    reasons = []
    if order.quantity <= 0:
        reasons.append(f"quantity must be > 0, got {order.quantity}")
    if order.unit_price < 0:
        reasons.append(f"unit_price must be >= 0, got {order.unit_price}")
    return reasons

Two rules, neither about product_id. This isn't a casual oversight — lesson 5 of foundations' module 5, in its own exercise 3, poses exactly this question ("Kiosko sells only its four known products... describe what new business rule you'd add") and leaves it as a reflection exercise, without building the code. The underlying reason is that checking product_id against a catalog is, technically, a different kind of check than quantity > 0: comparing against a fixed constant (0) is a local operation that doesn't need to look at anything outside the row itself; comparing against a catalog of valid products needs another data structure — a list or set of known IDs — that lives outside the row. It's exactly the boundary between validity (rules verified by looking only at the row) and consistency (rules that need to look at another table) that lesson 3 already named — and it's, precisely, the boundary this guide's module 3 formally crosses, with a real anti-join against dim_product.

It's also worth noting what check_business_rules() COULD, in theory, catch if someone decided to extend it: a unit_price of 60.00 doesn't break unit_price >= 0 — 60.00 is, without any doubt, greater than or equal to zero. No reasonable extension of "simple range rules" would catch that price without, again, an external baseline of what that product normally costs. That's the boundary between validity and accuracy, and it's why accuracy earns an entire module (module 5) later in this guide, instead of being "one more rule" tacked onto check_business_rules().

Common mistakes

Thinking that adding "one more rule" to check_business_rules() would solve consistency or accuracy. What happens: someone, seeing this table, proposes simply adding if order.product_id not in KNOWN_PRODUCT_IDS: reasons.append(...) inside check_business_rules(), as a minor extension of the existing function. Why it happens: it feels like the path of least effort — the function already exists, adding one more if seems trivial. How to spot it: if your implementation of the product_id check is a single line with an in/not in against a hand-coded fixed list, it works for Kiosko's case of three or four products, but it doesn't scale — it doesn't version the catalog, it doesn't sync automatically with dim_product in the warehouse, it doesn't distinguish "the product doesn't exist" from "the product exists but is discontinued." How to fix it: this guide builds consistency (module 3) as a real anti-join against the dim_product table read from kiosko.duckdb, not as a hand-coded list inside a row-validation function — it's a different architecture, not an extension of the existing one.

Concluding that validate_orders() "isn't good" because it lets problem rows through. What happens: after seeing this table, someone decides foundations' function is insufficient and should be discarded. Why it happens: it's easy to slide from "doesn't cover everything" to "isn't good for anything," without distinguishing the two. How to spot it: if your plan is to write a validate_orders_v2() from scratch for the rest of this guide, instead of keeping and extending the original, you missed this lesson's point. How to fix it: validate_orders() remains correct and useful for exactly what it does — completeness, validity, uniqueness. This guide doesn't replace it, it complements it with new pieces for the three dimensions it's missing; lessons 6 and 7 of this module run it, with no changes, against S04's first file.

Forgetting that this lesson's classification is a prediction, not yet an observation. What happens: someone memorizes this lesson's table as an already-proven fact, without having run validate_orders() against real S04 data yet. Why it happens: the table is presented with confidence, and it's easy to treat it as the final result instead of the starting point. How to spot it: if you can't explain which real file, with which exact rows, would confirm or refute this classification, you don't have evidence yet — you only have a well-founded hypothesis. How to fix it: lessons 6 and 7 of this module exist exactly for that — confirming this classification by running the real code against S04's first file, not just reasoning about the code in the abstract.

Exercises

Exercise 1 — Trace a hypothetical row through the four functions. Without running any code, trace by hand what would happen to this row if it went through validate_orders(): {"order_id": "ORD-Z1", "store_id": "S02", "product_id": "P888", "quantity": "5", "unit_price": "0.99", "order_ts": "2026-08-14T10:00:00"}. Does it end up in valid or in rejected? Why?

See solution

It ends up in valid. check_schema() finds no missing field — all six keys are present. check_nulls_and_types() finds no empty field, and all three conversions (int("5"), float("0.99"), datetime.fromisoformat(...)) work without a problem. check_business_rules() checks quantity <= 0 (5 <= 0 is false) and unit_price < 0 (0.99 < 0 is false) — no rule is violated. And order_id isn't in seen_order_ids (assuming this is its first appearance). The fact that P888 doesn't exist in PRODUCTS is completely invisible to all four functions — none of them looks at PRODUCTS at all. This row passes the entire gate with a perfectly clean bill of health, exactly the same mechanism that's going to let the real P099 row through in S04's file (lesson 7 of this module).

Exercise 2 — Confirm the classification by re-reading the source. Open (or re-read) data-engineering-foundations-guide, module 5, lesson 5 (check_business_rules()) and its Exercise 3. In 2-3 sentences, confirm that lesson already recognized the consistency gap, without resolving it.

See solution

That lesson's Exercise 3 literally asks: "Without writing code, describe in 2-3 sentences what new business rule you would add to check_business_rules() to catch a row with a product_id that doesn't exist in Kiosko's catalog, and explain why that rule belongs in this function and not in check_schema() or check_nulls_and_types()." That exercise's own solution acknowledges that rule would need to compare against PRODUCTS — a table external to the row —, exactly the kind of comparison this lesson classified as consistency, not validity. The gap isn't a new discovery of this guide — it's a question foundations left explicitly open, on purpose, for this guide to solve thoroughly.

Exercise 3 — Predict the result before running it. Without running any code yet — that's lesson 6's job —, predict: if you run validate_orders() against a file with six broken rows, one for each dimension from lesson 3's table (completeness, uniqueness, validity, consistency, freshness, accuracy), how many of those six would end up in rejected? Justify your number using this lesson's classification table.

See solution

Three of the six would end up in rejected: the completeness one, the uniqueness one, and the validity one — the three dimensions validate_orders() does check, according to this lesson's table. The other three — consistency, accuracy, and freshness — would pass through to valid with no warning at all, not because they're "less severe," but because none of validate_orders()'s four functions asks about them. Lesson 6 of this module confirms this exact prediction, running the real code against S04's first real file.

Summary and next step

In this lesson you precisely classified exactly which of the six data quality dimensions foundations' validate_orders() answers — completeness, validity, uniqueness — and confirmed, citing that guide's own exercise, that consistency had already been identified as a pending gap. You didn't change a single line of the original code — you brought it in, exact, and looked at it with lesson 3's new vocabulary.

Before moving on you should be able to: name the three dimensions validate_orders() does cover, and the three it doesn't; explain why check_business_rules() doesn't check product_id; and predict, without running code, what would happen to a row with a consistency or accuracy problem going through this gate.

You have the hypothesis. Now you need the real case to test it against. Lesson 5 introduces S04 Kiosko Reforma, Kiosko's fourth store — and tells, precisely, the story of its first failed contact with this very pipeline.

Resources

data-engineering-foundations-guide, module 5 (data-quality-gates) — the literal source of validate_orders() and the four functions that make it up. src/guides/data-engineering-foundations-guide/workbook/module-05-data-quality-gates/es/. In Spanish.
data-engineering-foundations-guide DESIGN — the original scope declaration for validate_orders(), the basis for this lesson's classification. src/guides/data-engineering-foundations-guide/DISENO.md. In Spanish.
Joe Reis & Matt Housley, Fundamentals of Data Engineering (O'Reilly, 2022) — the data quality framework that explicitly distinguishes schema validity from business-rule validity, already cited since foundations M5. oreilly.com/library/view/fundamentals-of-data/9781098108298. In English.

← PreviousThe six dimensions of data quality Next →Meet S04: Kiosko's fourth store

What foundations' `validate_orders()` already caught

Description

An analogy: the front-door guard, not the full auditor

Worked example: rebuilding validate_orders(), already familiar

The exact classification: what each piece asks

Diagram: the four pieces against the six dimensions

Going deeper: why check_business_rules() doesn't check product_id

Common mistakes

Exercises

Summary and next step

Resources

Worked example: rebuilding `validate_orders()`, already familiar

Going deeper: why `check_business_rules()` doesn't check `product_id`