Accuracy: the hardest dimension to test

Description

This guide's five earlier dimensions — completeness, uniqueness, validity, consistency, freshness (anticipated, not yet built) — share something this lesson makes explicit for the first time: each one has an objective answer, one that doesn't depend on any business judgment to decide. A field is empty or it isn't. An order_id repeats or it doesn't. Accuracy is different, and this lesson explains precisely why: it needs two human decisions — what's the "normal" value? how much deviation is acceptable? — before a single line of code can run.

Connection to the module. This lesson picks up directly from exercise 3 of module 1's lesson 3 — where it was already predicted, with no evidence yet, that accuracy would be the hardest — and confirms it with the full weight of the evidence accumulated across modules 1 through 4. Lesson 3 of this module goes deep into ORD-9509's concrete case; this lesson stays at the conceptual level, comparing the six dimensions against each other.

An analogy: the check, again, but looking closely at what each check asked

Lesson 1 of this module presented the perfectly filled-out check, for the wrong amount. It's worth returning to that same scene, but looking closely at the teller's full checklist, one item at a time, to notice something the first reading didn't make explicit: every item on that list — is it signed?, is the date valid?, does the numeric amount match the written-out amount? — has an answer the teller can confirm without leaving their window. They don't need to call anyone, they don't need to consult any external record, they don't need to exercise any judgment about whether the amount "makes sense" for that transaction. These are form checks, self-contained.

Now imagine the bank also wanted the teller to verify whether the check's amount "makes sense" for the customer issuing it — is it reasonable for this person to be paying this amount? That question can no longer be answered by looking at the check. It needs that account's transaction history, some criterion of "how much is reasonable" that someone at the bank had to decide beforehand, and an explicit tolerance — is a payment 20% above the historical average already suspicious, or does it need to be 200%? Accuracy is exactly that second question, applied to Kiosko's data: not "does the price have the correct form?", but "does the price make sense, compared to what that product normally costs?".

Worked example: the six dimensions, compared by what they need to decide

Module 1, lesson 3, already built a different check function for each of the six dimensions. It's worth running them again, this time paying attention to a new detail: how many external arguments — beyond the row itself — each one needs.

# what_each_check_needs.py
from datetime import datetime

KNOWN_PRODUCT_IDS = {"P001", "P002", "P003", "P004"}
PIPELINE_RUN_AT = "2026-08-16T09:00:00"  # this guide's fixed "now" -- never datetime.now()


def check_completeness(row: dict) -> bool:
    """Only needs the row itself."""
    return row["unit_price"] == ""


def check_uniqueness(order_id: str, seen: set[str]) -> bool:
    """Needs the set of IDs already seen IN THIS BATCH -- but no data external to the batch."""
    return order_id in seen


def check_validity(row: dict) -> bool:
    """Only needs the row itself, and a range rule fixed beforehand (quantity > 0)."""
    return row["quantity"] <= 0


def check_consistency(row: dict, known_ids: set[str]) -> bool:
    """Needs ANOTHER TABLE (the product catalog), but the question is binary: exists or not."""
    return row["product_id"] not in known_ids


def check_freshness(row: dict, run_at: str, sla_hours: int) -> bool:
    """Needs a fixed clock and an SLA -- but the SLA is a business rule already decided, not a range."""
    hours_elapsed = (datetime.fromisoformat(run_at) - datetime.fromisoformat(row["order_ts"])).total_seconds() / 3600
    return hours_elapsed > sla_hours


def check_accuracy(row: dict, reference: float, tolerance: float) -> bool:
    """Needs TWO business decisions: what the normal value is (reference),
    and how much deviation is acceptable (tolerance). Neither comes from the row."""
    return abs(row["unit_price"] - reference) / reference > tolerance


checks_needs = {
    "completeness": ["the row"],
    "uniqueness": ["the row", "IDs already seen (same batch)"],
    "validity": ["the row", "a range fixed beforehand"],
    "consistency": ["the row", "another table (catalog)"],
    "freshness": ["the row (or the file)", "a fixed clock", "an SLA already decided"],
    "accuracy": ["the row", "a baseline (reference)", "a tolerance"],
}

print(f"{'dimension':<14}{'what it needs, beyond the row'}")
for dim, needs in checks_needs.items():
    print(f"{dim:<14}{needs}")

What to expect. Running python3 what_each_check_needs.py, the output is exactly this:

dimension     what it needs, beyond the row
completeness  ['the row']
uniqueness    ['the row', 'IDs already seen (same batch)']
validity      ['the row', 'a range fixed beforehand']
consistency   ['the row', 'another table (catalog)']
freshness     ['the row (or the file)', 'a fixed clock', 'an SLA already decided']
accuracy      ['the row', 'a baseline (reference)', 'a tolerance']

Look at accuracy's list carefully, comparing it against the other five. consistency needs another table, that's true — but the question it answers with that table is binary and objective: product_id exists in dim_product, or it doesn't. There's no business judgment involved in deciding "what counts as existing" — either the row is in the table, or it isn't. freshness needs an SLA, but that SLA — 24 hours, in S04's case — is a single decision, made once, that afterward applies the same way to any file. accuracy is the only row in this table with two distinct business decisions, and neither one is binary: reference (is 1.20 the correct price, or should it be last week's price, or the average price across all stores?) and tolerance (is 12.5% deviation already suspicious, or does it need to be 50%?) are both numbers someone has to choose with judgment, not checks that can be derived mechanically from the data's structure.

Diagram: the spectrum of "how much business judgment each dimension needs"

flowchart LR
    subgraph OBJECTIVE["Objective -- a single mechanical check"]
        A["Completeness\nempty or not"]
        B["Validity\nwithin range or not"]
    end

    subgraph ONE_DECISION["One business decision, made once"]
        C["Uniqueness\nwhich field is the key"]
        D["Consistency\nwhich table is the source of truth"]
        E["Freshness\nwhat the SLA is"]
    end

    subgraph TWO_DECISIONS["Two business decisions, intertwined"]
        F["Accuracy\nwhat the normal value is\nAND how much deviation is acceptable"]
    end

    OBJECTIVE --> ONE_DECISION --> TWO_DECISIONS

The diagram orders the six dimensions on a spectrum, not a flat list: from left to right, each dimension needs progressively more human judgment before it can run. Accuracy isn't just further to the right — it's alone in its category, because the other five need, at most, a single decision (which SLA, which table is the source of truth), while accuracy needs two decisions that also interact with each other: changing the baseline changes which deviations look reasonable, and changing the tolerance changes how much imprecision in the baseline can be tolerated without generating noise.

Going deeper: why "harder" doesn't mean "impossible" or "hopelessly subjective"

It's worth cutting off, at the root, a wrong conclusion this lesson could unintentionally suggest: that accuracy, by needing human judgment, is a "subjective" dimension that can't be automated reliably. That's exactly the opposite of what the rest of this module builds. The real difference isn't "objective versus subjective" — it's where the decision lives. In completeness or validity, the decision (what counts as empty? what's the allowed range?) is so simple it nearly disappears inside the code itself: row["unit_price"] == "", quantity > 0. In accuracy, the decision is bigger and more visible — a calculated baseline, an explicit tolerance threshold — but it's still, once made, a completely mechanical, reproducible rule. check_price_baseline(), which this module builds starting in lesson 4, has absolutely nothing subjective in its execution — given the same reference_prices and the same tolerance, it always produces exactly the same result, on the same data. What changes, compared to check_validity(), is that the two decisions feeding it — the baseline, the tolerance — stay explicit and separate from the code, instead of hidden inside a trivial condition. That visibility is, in fact, an advantage: anyone can read reference_prices = {"P002": 1.20, ...} and ask whether that number is still correct, something you can't do as easily with an if quantity <= 0 buried inside a function.

Common mistakes

Concluding that, if accuracy needs "business judgment," then it can't be automated. What happens: someone, after reading that accuracy needs two human decisions, assumes the accuracy check has to be a manual, row-by-row review, done by a person. Why it happens: "business judgment" sounds, at first hearing, like the opposite of "automation." How to spot it: if your plan for accuracy involves someone reviewing every row by hand, you missed this lesson's central point in Going deeper. How to fix it: business judgment gets exercised once, when deciding reference_prices and tolerance — after that, check_price_baseline() runs in a completely mechanical, deterministic way over any number of rows, with no additional human intervention. It's exactly the same pattern freshness already used: the 24-hour SLA got decided once, and afterward check_freshness() (module 6) applies it with no additional judgment.

Thinking accuracy's difficulty is a flaw in this guide, something that "should" be solved with a fancier tool. What happens: someone, frustrated because accuracy needs more configuration work than the other five dimensions, looks for a library that "just detects" anomalous prices without anyone having to declare a baseline. Why it happens: after installing Pandera with a single pip install in module 2, expecting accuracy to be just as straightforward is a reasonable, though mistaken, expectation. How to spot it: if your search is "automatically detect incorrect prices with no configuration," you're already looking, without realizing it, for exactly the kind of Machine Learning model this module's lesson 1 declared out of scope. How to fix it: accept the dimension's nature — accuracy always needs a baseline declared by someone, human or model. This guide chooses for it to be a human, with a transparent formula, precisely to keep every decision explainable in one sentence.

Confusing "accuracy is hard to test" with "accuracy is the most important dimension." What happens: someone, impressed by the attention accuracy gets in this module, concludes it's "the dimension that matters most" of the six, above completeness or validity. Why it happens: dedicating a whole module to it feels like a signal of relative importance. How to spot it: ask yourself what would happen if Kiosko had no completeness check at all — a file with half its prices empty would break the pipeline far more immediately and visibly than a handful of slightly anomalous prices. How to fix it: "hard to test" and "more important" are different axes. Completeness and validity remain, in practice, the first lines of defense of any data quality system — accuracy is hard precisely because the problems it catches are more subtle, not because they're more severe than other dimensions' problems. The six dimensions are complementary, not a hierarchy of importance.

Exercises

Exercise 1 — Classify three new rules by how many business decisions they need. For each of these three hypothetical Kiosko rules, indicate whether it needs zero, one, or two business decisions before it can run (following this lesson's worked example's same criterion): (a) quantity can't be greater than 100 units in a single order; (b) store_id must exist in dim_store; (c) a store's daily revenue shouldn't deviate more than 40% from the average of the last 7 days.

Exercise 2 — Reproduce what_each_check_needs.py, and add a seventh hypothetical row. Kiosko decides to add a new rule: "every order must belong to a store that's open during order_ts's hours" (the same idea already suggested as an exercise in module 1, lesson 7). Add it to the worked example's checks_needs dictionary, deciding for yourself how many and which dependencies it needs.

checks_needs["store_hours"] = ["the row", "store hours table (new, doesn't exist in Kiosko yet)"]

Exercise 3 — Argue, in your own words, why tolerance is check_accuracy()'s more delicate piece, even more than reference. In 2-3 sentences, using the 1.35 versus 2.00 example module 1, lesson 3 (exercise 2) already worked through, explain why choosing tolerance badly can be more costly, in practice, than choosing the baseline badly.

Summary and next step

In this lesson you confirmed, comparing the six dimensions one by one, why accuracy is the hardest to test: not because it's "subjective" or impossible to automate, but because it needs two explicit business decisions — a baseline, a tolerance — before any code can run, while the other five need, at most, a single decision, almost always binary. You also made clear that this difficulty doesn't make it the "most important" of the six, nor an excuse to leave it unautomated.

Before moving on you should be able to: name the two business decisions accuracy needs, and explain why none of the other five dimensions needs both at once; and explain why "needs business judgment" isn't the same as "can't be automated."

Lesson 3 stays with ORD-9509's concrete case and dissects it completely: every check it passes, exactly why, and what would have to be true for a range rule to have, in theory, caught it — the last piece of the diagnosis before starting to build the solution in lesson 4.

Resources

• Module 1, lesson 3, of this same guide ("The six data quality dimensions") — the source of check_accuracy() and of the first prediction, with no evidence yet, that accuracy would be the hardest dimension. src/guides/data-reliability-and-governance-guide/workbook/module-01-when-green-does-not-mean-correct/en/03-six-dimensions-of-data-quality.md. In English.
• DAMA UK — "The Six Primary Dimensions for Data Quality Assessment" (October 2013) — the industry framework cited in this module's lesson 1. dama-uk.org/resources/the-six-primary-dimensions-for-data-quality-assessment. In English.
• Python — official datetime and timedelta documentation, reused with no changes in this lesson's worked example. docs.python.org/3/library/datetime.html. In English.
• This guide's DESIGN — module 5's complete map. src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.