Running the old gate on S04

Description

It's time to open the real file. This lesson presents the exact twelve lines of orders_2026-08-14.csv — S04's first sales file as a now-recognized store — and runs foundations' validate_orders(), with no changes, for real, against them. This isn't a hypothetical exercise or a prediction: you're going to see the literal, byte-for-byte output of running the same code you already built in data-engineering-foundations-guide.

Connection to the module. This lesson confirms, with real evidence, the prediction you built in lesson 4 (which dimensions validate_orders() covers) against the concrete case lesson 5 introduced (S04 and its first file). Lesson 7 takes this lesson's exact result and analyzes it thoroughly: what slipped through among the rows that passed.

The material: orders_2026-08-14.csv, twelve lines, fixed structure

Save this file exactly as it is — twelve data lines, none randomly generated — in the same folder as your kiosko.py:

order_id,store_id,product_id,quantity,unit_price,order_ts
ORD-9501,S04,P001,3,0.55,2026-08-14T08:05:00
ORD-9502,S04,P002,2,1.20,2026-08-14T08:12:00
ORD-9503,S04,P003,1,,2026-08-14T08:19:00
ORD-9504,S04,P004,1,4.50,2026-08-14T08:27:00
ORD-9505,S04,P001,2,0.55,2026-08-14T08:34:00
ORD-9506,S04,P003,3,0.75,2026-08-14T08:41:00
ORD-9507,S04,P001,-1,0.55,2026-08-14T08:49:00
ORD-9508,S04,P099,2,1.00,2026-08-14T08:56:00
ORD-9509,S04,P002,1,60.00,2026-08-14T09:04:00
ORD-9502,S04,P002,2,1.20,2026-08-14T09:11:00
ORD-9510,S04,P004,2,4.50,2026-08-14T09:18:00
ORD-9511,S04,P002,1,1.20,2026-08-14T09:25:00

Twelve lines, all twelve from S04, none of the other three stores mixed in. Before running anything, it's worth looking at the file with the trained eyes from lessons 2 through 5 — not to guess the result, but to practice the critical reading this entire guide trains:

• ORD-9503 has a double comma at unit_price (P003,1,,2026-08-14...) — the field is present in the schema, but empty.
• ORD-9507 has quantity=-1 — a negative value where only a positive integer makes sense.
• ORD-9508 has product_id=P099 — a product that doesn't appear in Kiosko's four-product catalog (P001 through P004).
• ORD-9509 has product_id=P002 with unit_price=60.00 — sixty dollars for an Energy Bar.
• ORD-9502 appears twice: first on the file's second line, and again on the tenth — the same order_id, the same store, the same product, the same quantity, the same price, seven minutes apart in order_ts — the exact pattern of a duplicate retransmission, not two separate sales.

Five named problems, across six physical rows (ORD-9502 counts twice, once per appearance) — exactly the twelve-line structure, six clean and six broken, one per quality dimension, that defines this guide. You already confirmed the sixth dimension — freshness — in lesson 5: it's a property of the whole file, not of any individual row, so it doesn't show up as a "flagged" line inside this CSV.

Worked example: validate_orders(), actually run against S04

# diagnose_s04.py
import csv
from kiosko import validate_orders, print_validation_report

with open("orders_2026-08-14.csv", newline="") as f:
    rows = list(csv.DictReader(f))

print(f"Rows read from orders_2026-08-14.csv: {len(rows)}\n")

valid, rejected = validate_orders(rows)
print_validation_report(valid, rejected)

What to expect. Running python3 diagnose_s04.py in the folder where you saved orders_2026-08-14.csv (with lesson 4's kiosko.py in the same folder), the output is exactly this:

Rows read from orders_2026-08-14.csv: 12

=== Kiosko: validation report ===
Total rows: 12
Valid: 9
Rejected: 3

=== Rejected row details ===
ORD-9503:
  - field 'unit_price' is null or empty
ORD-9507:
  - quantity must be > 0, got -1
ORD-9502:
  - duplicate order_id 'ORD-9502'

Read this result with the same care you already trained in foundations M5: twelve rows read, nine end up valid, three end up rejected, each with its exact reason. ORD-9503 gets rejected for an empty unit_price — exactly the check_nulls_and_types() check you already know, covering completeness. ORD-9507 gets rejected for quantity must be > 0, got -1 — check_business_rules(), covering validity. And ORD-9502 gets rejected for duplicate order_id — but notice an important detail: only one of the two appearances of ORD-9502 shows up in rejected. The first (line 2 of the file, 08:12:00) passed with no problem, because at that point seen_order_ids didn't know about it yet. The second (line 10, 09:11:00) is the one that gets flagged — exactly the same behavior you already confirmed with ORD-8001 in foundations M5's project.

The detail that matters: what stayed inside valid

Valid: 9 with no additional detail isn't enough to diagnose anything — it's exactly the kind of bare count foundations M5's lesson 7 already warned against. It's worth looking, row by row, at what those nine actually are:

# inspect_valid.py -- added to the end of diagnose_s04.py
print("\n=== The 9 rows that stayed in 'valid' ===")
for row in valid:
    print(f"  {row['order_id']} | product_id={row['product_id']} | unit_price={row['unit_price']} | quantity={row['quantity']}")

What to expect. Adding this block to the end of diagnose_s04.py and running it again, after the already-familiar report, the additional output is exactly this:

=== The 9 rows that stayed in 'valid' ===
  ORD-9501 | product_id=P001 | unit_price=0.55 | quantity=3
  ORD-9502 | product_id=P002 | unit_price=1.20 | quantity=2
  ORD-9504 | product_id=P004 | unit_price=4.50 | quantity=1
  ORD-9505 | product_id=P001 | unit_price=0.55 | quantity=2
  ORD-9506 | product_id=P003 | unit_price=0.75 | quantity=3
  ORD-9508 | product_id=P099 | unit_price=1.00 | quantity=2
  ORD-9509 | product_id=P002 | unit_price=60.00 | quantity=1
  ORD-9510 | product_id=P004 | unit_price=4.50 | quantity=2
  ORD-9511 | product_id=P002 | unit_price=1.20 | quantity=1

Stop on two of these nine lines before moving on. ORD-9508, with product_id=P099 — you already know, because you read it in this lesson's "The material" section, that P099 doesn't exist in Kiosko's four-product catalog. And yet, there it is, inside valid, with no flag, no rejection reason attached. ORD-9509, with unit_price=60.00 for an Energy Bar (P002) — the same product that, in the other three appearances in this very file (ORD-9502 twice, ORD-9511), sold for 1.20. A difference of fifty times the usual price, within the same file, for the same product, on the same day — and also with no flag. These two rows are the exact confirmation, with real data, of the prediction you built in lesson 4: validate_orders() never asked anything about referential consistency or anomalous prices, so these two rows pass with the same clean bill of health as any perfectly correct row.

Diagram: the twelve rows, classified by actual outcome

flowchart TD
    A["12 rows from orders_2026-08-14.csv"] --> B["validate_orders()"]
    B --> C["rejected: 3 rows"]
    B --> D["valid: 9 rows"]

    C --> C1["ORD-9503: completeness"]
    C --> C2["ORD-9507: validity"]
    C --> C3["ORD-9502 (2nd appearance): uniqueness"]

    D --> D1["6 genuinely clean rows"]
    D --> D2["ORD-9502 (1st appearance):\npart of the duplicate pair,\nbut THIS ONE is valid"]
    D --> D3["ORD-9508: product_id=P099\n(consistency, unflagged)"]
    D --> D4["ORD-9509: unit_price=60.00\n(accuracy, unflagged)"]

The diagram splits the nine-row valid block into three categories: the six genuinely correct ones, ORD-9502's first appearance (which is valid on its own — the problem is that it repeats, not that it, individually, is wrong), and the two rows with real problems the gate doesn't detect. Notice that, from validate_orders()'s perspective, the nine valid rows are indistinguishable from each other — the function has no way of telling you "these six are genuinely fine, while these other three only passed because I never asked the right question." That's, precisely, the problem lesson 7 analyzes in depth.

Going deeper: why the 9/12 count is, on its own, already incomplete information

Someone who only looks at Valid: 9, Rejected: 3 — without ever printing valid's detail, as this lesson did — gets a reasonably reassuring impression: 75% of the rows passed clean. That impression is mathematically correct and, at the same time, misleading in the most important sense: of those nine "clean" rows, two have real business problems — a nonexistent product, a price off by two orders of magnitude — that no number in print_validation_report()'s report reveals. The report isn't lying; it simply wasn't designed to ask the questions that would reveal those two problems.

This is, with a real, executed case, the lie of the green checkmark this module's lesson 2 named: Valid: 9 is an honest fact about how many rows passed the four checks validate_orders() knows how to run. It isn't, and doesn't claim to be, a statement about whether those nine rows are correct in the broader business sense. The gap between those two claims — "passed the checks that exist" versus "is correct" — is exactly the space the rest of this guide fills, module by module.

Common mistakes

Stopping at Rejected: 3 and never checking valid's content. What happens: someone runs diagnose_s04.py, sees the rejection report, and considers the file's diagnosis complete without ever printing or inspecting the nine valid rows. Why it happens: print_validation_report() already feels like "the complete report" — it's easy to forget it only describes the rejected half, never the accepted one. How to spot it: if your summary of S04's file is "three broken rows, nine clean ones," with no further nuance, you're missing this lesson's central observation — two of those nine "clean" rows have real problems. How to fix it: any time you use validate_orders() on data you don't know thoroughly, inspect valid too, not just rejected — this lesson's code block that prints every valid row is exactly that habit.

Being surprised that ORD-9502 doesn't appear twice in rejected. What happens: someone expects to see both appearances of ORD-9502 in the rejected-rows detail, and gets confused seeing only one. Why it happens: intuitively, if an order_id is "duplicated," it feels like both copies should be treated the same way. How to spot it: go back to validate_orders()'s flow diagram in foundations M5 (lesson 6 of that module) — the duplicate check compares against seen_order_ids, which fills up while the function processes rows in order. The first appearance has nothing to compare against yet. How to fix it: remember the exact rule, already established in foundations: the function always flags the later appearance as the duplicate, never the first — the same behavior you already saw with ORD-8001 in that guide, now confirmed again with ORD-9502.

Blaming the CSV file for having "badly written data." What happens: someone describes orders_2026-08-14.csv as a file with formatting or writing errors, as if the problem were syntactic. Why it happens: twelve lines with six problems feels, at first glance, like "a careless file." How to spot it: check each of the six problem rows from this lesson's "The material" section — none has a CSV syntax error (mismatched quotes, extra or missing columns). Each one is syntactically perfect and, even so, has a content problem: an empty field, an out-of-range value, a nonexistent reference, an anomalous price, a retransmission. How to fix it: always distinguish between "the file is malformed" (a syntax problem, which not even csv.DictReader could read) and "the file is well-formed but has incorrect content" (this lesson's real problem, and this entire guide's) — they're completely different categories of failure.

Exercises

Exercise 1 — Confirm the count of broken rows per dimension, by hand. Without running any code again, count how many of this lesson's twelve rows from "The material" correspond to each of lesson 3's six quality dimensions (including freshness, even though it isn't an individual row). Confirm that the total number of "problem" rows (not counting freshness, which belongs to the whole file) is six.

Exercise 2 — Rewrite the report to count duplicates without double-counting. print_validation_report()'s report counts ORD-9502 only once in rejected (the second appearance). Write a small script that counts how many distinct order_ids appear across the file's twelve rows, and compare it against the total row count (12).

order_ids = [row["order_id"] for row in rows]
distinct_order_ids = set(order_ids)
print(f"Total rows: {len(rows)}")
print(f"Distinct order_id: {len(distinct_order_ids)}")

Total rows: 12
Distinct order_id: 11

Exercise 3 — Argue why ORD-9502 (first appearance) should NOT be considered "suspicious" on its own. In 2-3 sentences, explain why it would be a mistake to treat ORD-9502's first appearance — the one that stays in valid — as if it, individually, had some quality problem.

Summary and next step

In this lesson you actually ran validate_orders(), with no changes, against the twelve real lines of orders_2026-08-14.csv: nine valid rows, three rejected, each with its exact reason. And, going beyond the bare count, you inspected the content of those nine "valid" rows and confirmed, with direct evidence, that two of them — ORD-9508 (P099) and ORD-9509 (60.00) — have real business problems that none of the gate's four checks could detect.

Before moving on you should be able to: reproduce this lesson's exact report by running the code yourself; explain why ORD-9502 appears only once in rejected, not twice; and name, without looking at the code again, which two valid rows have real problems without being flagged.

You have the complete evidence. Lesson 7 analyzes those two silent rows in depth — why exactly they slip past validate_orders(), and what kind of tool would need to exist to catch them — closing this module's full diagnosis.

Resources

• data-engineering-foundations-guide, module 5 (data-quality-gates) — the source of validate_orders(), run with no changes in this lesson. src/guides/data-engineering-foundations-guide/workbook/module-05-data-quality-gates/es/. In Spanish.
• Python — official csv.DictReader documentation, the reading tool used in the worked example. docs.python.org/3/library/csv.html. In English.
• Joe Reis & Matt Housley, Fundamentals of Data Engineering (O'Reilly, 2022) — the data quality framework that explains why an aggregate count never replaces inspecting the real content. oreilly.com/library/view/fundamentals-of-data/9781098108298. In English.