Module introduction: accuracy and deterministic anomaly detection

Why this module exists

Four modules, and ORD-9509 is exactly where it started: with no flag at all. Review it with the same precision this guide reviews everything else. Module 1 ran foundations' validate_orders() over the twelve rows of orders_2026-08-14.csv and left it in valid, alongside the other eight genuinely clean rows — "ORD-9509 | product_id=P002 | unit_price=60.00 | quantity=1", printed with no distinguishing mark among the correct rows. Module 2 wrote OrdersSchema with Pandera — completeness, uniqueness, validity, declared instead of hand-coded — and its closing project reported 8 of 12 rows passing with no error, with ORD-9509 inside that list again, now with the float intact: unit_price=60.0. Module 3 added referential integrity — validate_referential_integrity(), the anti-join that finally caught ORD-9508 (product_id="P099") — and its final report named, with the honesty of a good auditor, exactly what remained pending: DIMENSIONS_STILL_OPEN = ["accuracy", "freshness"], printed, literally, at the end of that project. Module 4 turned those same rules into a versioned data contract, orders_contract.yaml, generated OrdersSchema from YAML instead of Python — and ORD-9509 was still in the list of "order_id with no known problem under this contract," eight names, its own among them.

None of those four tools failed. Each did exactly what it was designed to do, and did it well. The problem runs deeper than "a tool with a bug": 60.00 is a positive number, of type float, within any reasonable range someone might have declared for a price. There's no schema, no Field, no type or range Check that can distinguish that value from a genuine price — because, looked at in isolation, there's nothing to distinguish. The error doesn't live in the row. It lives in the relationship between the row and what that product normally costs, and none of the four earlier modules' tools has any parameter through which that relationship could enter.

This module closes that gap, and with it closes the complete diagnosis module 1 opened: accuracy, the sixth dimension, the only one of the six that needs something no range rule can derive on its own — an external baseline, calculated over data already confirmed reliable, and a tolerance threshold that decides how much deviation is normal business and which is a real alarm.

Connection to the previous module. Module 4 said it in the same words that open this module: "module 5 doesn't rewrite any schema from scratch: it builds a completely new tool — a price baseline — because accuracy is, precisely, the kind of rule no declarative schema, whether hand-written or generated from a contract, can express." This module fulfills that promise, word for word.

A bit of history: accuracy is not an afterthought

The distinction between "valid" and "correct" isn't a discovery of this guide — it's one of the six primary dimensions the data management industry already formalized over a decade ago. The DAMA UK working group (the British branch of the world's largest professional data management association) published, in October 2013, "The Six Primary Dimensions for Data Quality Assessment" — the same six-dimension framework (accuracy, completeness, consistency, timeliness, uniqueness, validity) this guide has already used since module 1, with one minor naming difference (this guide uses freshness where DAMA UK uses timeliness, the same concept). The document defines accuracy as how well a piece of data reflects the real-world object or event it describes, and it's explicit that evaluating it requires comparing the data against an authoritative reference source — exactly the piece this module builds, reference_prices, calculated over the only portion of Kiosko's data already confirmed, across eight earlier guides in this ecosystem, as reliable: the canonical week.

It's worth saying it with the same honesty this guide's market warning already demanded: completeness, uniqueness, and validity are, in practice, relatively cheap to automate — a range rule, a null check, a type comparison. Accuracy, by contrast, demands business judgment before writing a single line of code: what's the "normal" price? how much deviation is a legitimate promotion, and how much is an error? No framework, not Pandera, not Great Expectations, not Soda, can make that decision for Kiosko — it can only execute the decision, once someone has made it.

An analogy: the perfectly filled-out check, for the wrong amount

Think of a bank check. A teller who receives it reviews a list of formal requirements: is it signed? is the date valid? does the numeric amount match the written-out amount? isn't it torn or visibly altered? A check that passes all those checks is, in the strictest sense, a valid check — it has the correct form, every required field is present and well-formed. And yet, that same check, perfectly valid, could be paying five hundred dollars for a five-dollar purchase — a typing error, an extra zero, a misplaced decimal separator. None of the formal requirements on the teller's list detects that error, because the error doesn't live in the check's form. It lives in whether the number, however perfectly written, corresponds to the reality of what's being paid.

ORD-9509 is that check. unit_price=60.00 is perfectly formed: it's a number, it's positive, it fits in any DOUBLE column. And it's paying sixty dollars for an energy bar that, in every other appearance of that same product in that same file, cost 1.20. The teller in the analogy — any of the four tools from earlier modules — has no way to detect that error by looking only at the check. It needs something more: knowing, beforehand, how much what's being paid normally costs. That "knowing beforehand" is, precisely, what this module builds.

Worked example: reproducing the accumulated evidence, in one place

Before building anything new, it's worth confirming, with a single run, the fact that sustains this entire module: that ORD-9509's isolated row passes, with no exception at all, the most complete tool that already exists in this guide — Pandera's OrdersSchema, with the three rules from modules 2 and 4 already complete.

# ord_9509_alone.py
import polars as pl
import pandera.polars as pa


class OrdersSchema(pa.DataFrameModel):
    order_id: str = pa.Field(unique=True)
    unit_price: float = pa.Field(nullable=False, ge=0)
    quantity: int = pa.Field(gt=0)


# ORD-9509, isolated, exactly as it appears in orders_2026-08-14.csv
ord_9509 = pl.DataFrame({
    "order_id": ["ORD-9509"],
    "store_id": ["S04"],
    "product_id": ["P002"],
    "quantity": [1],
    "unit_price": [60.00],
})

validated = OrdersSchema.validate(ord_9509, lazy=True)
print("OrdersSchema.validate(ord_9509, lazy=True) -- no exception at all")
print(validated)

What to expect. Running python3 ord_9509_alone.py, the output is exactly this:

OrdersSchema.validate(ord_9509, lazy=True) -- no exception at all
shape: (1, 5)
┌──────────┬──────────┬────────────┬──────────┬────────────┐
│ order_id ┆ store_id ┆ product_id ┆ quantity ┆ unit_price │
│ ---      ┆ ---      ┆ ---        ┆ ---      ┆ ---        │
│ str      ┆ str      ┆ str        ┆ i64      ┆ f64        │
╞══════════╪══════════╪════════════╪══════════╪════════════╡
│ ORD-9509 ┆ S04      ┆ P002       ┆ 1        ┆ 60.0       │
└──────────┴──────────┴────────────┴──────────┴────────────┘

Not a single row discarded, no exception, no SchemaError. order_id is unique (in a single-row DataFrame, trivially), unit_price is not null and is >= 0, quantity is > 0. The three completeness, uniqueness, and validity rules this ecosystem already built, applied to the single most problematic row in S04's entire incident, find absolutely nothing wrong — because, in the strict sense in which they're written, nothing is wrong. 60.00 is, for any of those three rules, as valid a price as 1.20. The only way to tell them apart is knowing, beforehand, what an Energy Bar's normal price is — and that knowledge doesn't live, and can't live, inside nullable, unique, or gt.

Diagram: four modules' accumulated diagnosis

flowchart TD
    A["ORD-9509\nP002 / unit_price=60.00"] --> B["Module 1:\nvalidate_orders()\nIN valid, no flag"]
    A --> C["Module 2:\nOrdersSchema (Pandera)\nIN passing, no flag"]
    A --> D["Module 3:\nvalidate_referential_integrity()\nP002 DOES exist -- no flag\n(the right tool,\nthe wrong question)"]
    A --> E["Module 4:\ncontract_to_pandera_schema()\nIN 'no known problem', no flag"]
    B --> F["Four tools.\nZero flags.\nThe problem remains intact."]
    C --> F
    D --> F
    E --> F
    F --> G["Module 5 (this one):\nreference_prices +\ncheck_price_baseline()"]
    G --> H["ORD-9509: ANOMALOUS\n60.00 is 50x the reference\nprice (~1.20)"]

The diagram is deliberately repetitive — four different arrows, the same result four times. That repetition is the point: it isn't that one tool forgot to check something. It's that all four tools, correctly designed for what they were meant to solve, share an identical structural limitation against this specific row.

The map of this guide's 8 modules (reminder)

The map of this module

Lesson    Question it answers
────────  ──────────────────────────────────────────────────────────────
L1        (this one) Why, after four modules, ORD-9509 still has no
          flag at all -- and why no earlier tool failed.
L2        Why accuracy is, with evidence, the hardest of the six
          dimensions to test.
L3        ORD-9509 in depth: every check it passes, and why "valid"
          and "correct" stop being synonyms here.
L4        Building reference_prices from Kiosko's clean canonical
          week (S01-S03) -- never from the file under suspicion.
L5        check_price_baseline(): threshold-based anomaly detection,
          with no Machine Learning, tested first on toy data.
L6        The same function, run on real S04: ORD-9509, finally
          caught, with executed evidence.
L7        What happens if tolerance is miscalibrated -- too strict
          (false positives) or too loose (false negatives).
L8        Project: S04's complete accuracy audit.

Lessons 2 and 3 complete the diagnosis: why accuracy is different from the other five dimensions, and what exactly makes ORD-9509 invisible to any range rule. Lesson 4 is where the real construction begins — the baseline, calculated over the only portion of Kiosko's data eight earlier guides already confirmed reliable. Lesson 5 writes the detection function, first on toy data, following the same pedagogical order module 3 already used with validate_referential_integrity(). Lesson 6 is this module's central moment: the same function, with no change at all, run on the real file — the dollars-to-cents bug, caught. Lesson 7 confronts a question every threshold faces sooner or later: how sensitive should it be? And lesson 8, the project, assembles everything into S04's complete accuracy audit.

The boundary: what does NOT enter this module

There's an ecosystem boundary worth drawing from the start, because it's going to come up again in every lesson of this module: anomaly detection with Machine Learning — statistical time-series models, unsupervised detection, any technique that learns a pattern from the data instead of a human declaring it explicitly — is outside the scope of NIEVA Data Engineering. This is an ecosystem decision, not a technical limitation of this guide: NIEVA doesn't train data scientists, it trains data engineers, and the specialty of anomaly detection with machine learning — models like Isolation Forest, ARIMA, or neural networks trained on historical series — belongs to another discipline, with another set of guides.

Everything this module builds is deterministic: a rule, a threshold, a reason explainable in a single sentence. check_price_baseline() doesn't "learn" what price is normal — someone (this guide) decides, with a transparent formula, AVG(unit_price) over data already confirmed clean; and someone decides, also with a transparent formula, how much percentage deviation triggers an alert. If ORD-9509 gets flagged as anomalous, anyone at Kiosko — not just whoever wrote the code — can read the exact reason: "60.00 deviates 49 times (4900%) from the reference price of 1.20, beyond the 50% configured tolerance." No black-box model, however precise, offers that same clarity without additional interpretability work — and this guide, consistent with the eight earlier ones in the ecosystem, prioritizes that clarity over any marginal precision gain a statistical model might contribute on a data case of this size.

Common mistakes

Thinking this module is going to "fix" validate_orders(), OrdersSchema, or module 4's contract. What happens: someone, after reading this lesson's diagnosis, expects the next lessons to modify one of the four already-built tools so it also catches accuracy. Why it happens: it seems natural that the solution to "this tool doesn't catch it" would be "modify this tool." How to spot it: check this module's central function's name, check_price_baseline() — it's a completely new function, not a parameter added to any earlier one. How to fix it: recall module 1, lesson 7's exact deep-dive: every dimension that needs an external reference earns its own piece, instead of bloating a function that already served its original purpose. validate_orders(), OrdersSchema, and module 4's contract aren't going to change in this module — they're going to stay exactly as they are, and a new tool gets added alongside them.

Expecting this module to introduce some kind of statistical or Machine Learning model. What happens: someone, familiar with the term "anomaly detection" from other contexts (infrastructure monitoring, fraud detection), expects to see something like a time-series model or a clustering algorithm. Why it happens: in many industries, "anomaly detection" is synonymous with Machine Learning. How to spot it: if your expectation includes words like "train," "model," "prediction," you're already outside the scope this lesson's boundary section drew. How to fix it: everything this module builds is a simple arithmetic formula — a percentage difference against a fixed baseline — compared against a fixed threshold. It's "anomaly detection" in the most literal, least sophisticated sense of the term: a rule anyone can read and verify by hand, with no black box involved.

Believing ORD-9509 is a rare case, not representative of real data problems. What happens: someone sees the specific error — sixty dollars instead of sixty cents — and treats it as a toy example, unlikely in a real production system. Why it happens: 60.00 instead of 0.60 sounds, at first glance, like an error invented for a lesson. How to spot it: review the market warning that opens this guide's complete design — the dollars-to-cents bug is, literally, one of the incidents cited by real market evidence gathered from practitioners, not a hypothetical case. How to fix it: treat this incident with the seriousness it deserves — a data source that switches units (cents instead of dollars, or the other way around) with no warning at all is exactly the kind of silent failure that occurs in real production systems, with real financial consequences, precisely because it passes every schema check with no problem at all.

Exercises

Exercise 1 — Reproduce the worked example with a second "suspicious" row, chosen by you. Using ord_9509_alone.py's same pattern, build a single-row DataFrame with product_id="P001" and unit_price=0.01 (one cent for a bottle of water). Run OrdersSchema.validate() on it. Does it pass or fail? Why?

suspicious_row = pl.DataFrame({
    "order_id": ["ORD-X1"],
    "store_id": ["S04"],
    "product_id": ["P001"],
    "quantity": [1],
    "unit_price": [0.01],
})

validated = OrdersSchema.validate(suspicious_row, lazy=True)
print(validated)

Exercise 2 — List, without looking back, the four tools that have already passed ORD-9509. From memory, name the four tools from modules 1 through 4 that each, on its own, already confirmed ORD-9509 has no known problem. For each, in one sentence, explain why.

Exercise 3 — Argue why this lesson's Machine Learning boundary isn't just a limitation of this guide, but a defensible design decision. In 3-4 sentences, and using this lesson's check analogy, argue why a deterministic, explainable rule — "60.00 deviates 4900% from the reference price of 1.20" — can be preferable, in a business context like Kiosko's, to a more sophisticated statistical model that only said "anomaly score: 0.97" with no readable explanation at all.

Summary and next step

In this lesson you confirmed, with executed evidence, the exact problem this module solves: ORD-9509, with unit_price=60.00, sailed cleanly through the four quality tools this guide already built — validate_orders(), OrdersSchema, validate_referential_integrity(), module 4's generated contract — not because any of them failed, but because none of them has a parameter through which a price baseline could enter. You learned about the DAMA UK framework that has formalized accuracy as one of the six primary data quality dimensions since 2013, and you drew, from the module's start, the Machine Learning boundary that's going to sustain every lesson that follows.

Before moving on you should be able to: explain, citing this lesson's executed result, why OrdersSchema doesn't catch ORD-9509; name the four tools from modules 1 through 4 and why each one, correctly, let it through; and explain in your own words why this module deliberately excludes any Machine Learning technique.

Lesson 2 goes deeper into the question this introduction already started answering: of the six data quality dimensions, why is accuracy, with evidence consistent across this entire guide, the hardest to test?

Resources

• DAMA UK — "The Six Primary Dimensions for Data Quality Assessment" (whitepaper, October 2013, DAMA UK Working Group) — the industry framework that formalizes accuracy as one of the six primary data quality dimensions, evaluated against an authoritative reference source. dama-uk.org/resources/the-six-primary-dimensions-for-data-quality-assessment. In English.
• Pandera — official documentation (DataFrameModel, Field, the complete reference for the completeness/uniqueness/validity rules already built in modules 2 and 4). pandera.readthedocs.io. In English.
• Module 1, lesson 7, of this same guide ("What slips through a local gate") — the literal source of check_price_baseline() as an already-anticipated name, and of the critique of comparing a file against its own median instead of against the canonical week. src/guides/data-reliability-and-governance-guide/workbook/module-01-when-green-does-not-mean-correct/en/07-what-slips-through-a-local-gate.md. In English.
• Module 3, project (lesson 8), and module 4, project (lesson 8), of this same guide — the source of the accumulated evidence cited in this lesson (DIMENSIONS_STILL_OPEN, the list of order_id with no known problem). src/guides/data-reliability-and-governance-guide/workbook/module-03-consistency-and-referential-checks/en/08-project-s04s-full-consistency-report.md and src/guides/data-reliability-and-governance-guide/workbook/module-04-data-contracts-as-versioned-artifacts/en/08-project-kioskos-first-data-contract.md. In English.
• This guide's DESIGN — this module 5's exact mandate, including the Machine Learning boundary. src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.