The six dimensions of data quality

Description

"The data is bad" isn't a useful sentence. It doesn't tell anyone what to check, how severe the problem is, or whether two people using that phrase are even talking about the same kind of failure. This lesson replaces that vague phrase with six precise words — completeness, uniqueness, validity, consistency, freshness, accuracy — each with an exact definition and a runnable example. From here on, throughout this entire guide, "the data is bad" always turns into one of these six specific claims, never a generic complaint.

Connection to the module. This vocabulary is the foundation for lessons 4 through 8 of this module, and for the entire guide: lesson 4 classifies which dimensions foundations' validate_orders() covers; lessons 6 and 7 run that function against S04's first real file and classify, dimension by dimension, what it catches and what it doesn't; and each following module in this guide — Pandera (M2), consistency (M3), contracts (M4), anomalies (M5), freshness and lineage (M6) — thoroughly solves one or two of these six dimensions. Without this precise vocabulary, the rest of the guide would be a list of tools with no thread connecting them.

An analogy: a restaurant inspector's checklist

A health inspector visiting a restaurant doesn't ask "does it look good?" — that question doesn't help anyone decide whether the restaurant is safe. Instead, they work through a checklist of concrete, verifiable items one by one: is the food stored at the right temperature? Do employees wash their hands? Are expiration dates legible? Is the cleaning log complete, with no missing days? Each item on that list is a specific question, with a binary or measurable answer, not a general impression.

The six dimensions of data quality are that same checklist, applied to a table instead of a kitchen. Instead of asking "is the data okay?" — a question as vague as "does it look good?" —, a data reliability engineer asks, one at a time: is every required field complete? Does every identifier appear exactly once? Does every value respect its correct type and range? Does every reference to another table actually point to something that exists? Did the data arrive on time? Does the value, even if technically valid, reflect reality? Six concrete questions, each verifiable on its own, instead of one blurry impression.

Worked example: the six dimensions, one broken row per dimension

Each of this example's six rows has, on purpose, exactly one problem — none has two at once, so each dimension stays isolated and easy to recognize:

# six_dimensions.py
from datetime import datetime

KNOWN_PRODUCT_IDS = {"P001", "P002", "P003", "P004"}
REFERENCE_PRICE_P002 = 1.20  # P002's usual price in Kiosko's canonical week
PIPELINE_RUN_AT = "2026-08-16T09:00:00"  # this guide's fixed "now" -- never datetime.now()

completeness_row = {"order_id": "ORD-D1", "unit_price": ""}
uniqueness_seen = {"ORD-D2"}  # order_id already processed earlier in the same batch
uniqueness_row_id = "ORD-D2"
validity_row = {"order_id": "ORD-D3", "quantity": -1}
consistency_row = {"order_id": "ORD-D4", "product_id": "P099"}
freshness_row = {"order_id": "ORD-D5", "order_ts": "2026-08-14T09:25:00"}
accuracy_row = {"order_id": "ORD-D6", "product_id": "P002", "unit_price": 60.00}


def check_completeness(row: dict) -> bool:
    """True if the required field is empty -- violates completeness."""
    return row["unit_price"] == ""


def check_uniqueness(order_id: str, seen: set[str]) -> bool:
    """True if the order_id already appeared before -- violates uniqueness."""
    return order_id in seen


def check_validity(row: dict) -> bool:
    """True if the value breaks a range rule -- violates validity."""
    return row["quantity"] <= 0


def check_consistency(row: dict) -> bool:
    """True if the product_id doesn't exist in the catalog -- violates consistency."""
    return row["product_id"] not in KNOWN_PRODUCT_IDS


def check_freshness(row: dict, run_at: str, sla_hours: int) -> bool:
    """True if more time than the SLA passed between the order and when the check runs -- violates freshness."""
    hours_elapsed = (datetime.fromisoformat(run_at) - datetime.fromisoformat(row["order_ts"])).total_seconds() / 3600
    return hours_elapsed > sla_hours


def check_accuracy(row: dict, reference: float, tolerance: float) -> bool:
    """True if the value strays too far from the baseline -- violates accuracy."""
    return abs(row["unit_price"] - reference) / reference > tolerance


results = [
    ("completeness", "ORD-D1", check_completeness(completeness_row)),
    ("uniqueness", "ORD-D2", check_uniqueness(uniqueness_row_id, uniqueness_seen)),
    ("validity", "ORD-D3", check_validity(validity_row)),
    ("consistency", "ORD-D4", check_consistency(consistency_row)),
    ("freshness", "ORD-D5", check_freshness(freshness_row, PIPELINE_RUN_AT, 24)),
    ("accuracy", "ORD-D6", check_accuracy(accuracy_row, REFERENCE_PRICE_P002, 0.5)),
]

print(f"{'dimension':<14}{'order_id':<10}{'violates?'}")
for dim, oid, violates in results:
    print(f"{dim:<14}{oid:<10}{violates}")

What to expect. Running python3 six_dimensions.py, the output is exactly this:

dimension     order_id  violates?
completeness  ORD-D1    True
uniqueness    ORD-D2    True
validity      ORD-D3    True
consistency   ORD-D4    True
freshness     ORD-D5    True
accuracy      ORD-D6    True

All six rows violate their corresponding dimension, each for a different reason and verified with a different function. Notice that none of the six check functions looks like the others: check_completeness() asks whether a field is empty; check_uniqueness() asks whether an identifier was already seen; check_validity() asks whether a value respects a range; check_consistency() asks whether a reference exists in another table; check_freshness() compares two points in time; check_accuracy() compares a value against a reference baseline. Six questions, six different mechanics — and that's why no single tool solves all six at once without thinking through them separately, as you'll see throughout this guide.

The six dimensions, defined precisely

Notice the "Level" column: the first four dimensions — completeness, uniqueness, validity, consistency — are verified row by row (although uniqueness and consistency need to look at something beyond the row alone: a set of already-seen IDs, or an external table). Freshness, on the other hand, is a property of the whole file or table — it doesn't make sense to ask "is this individual row fresh?"; the right question is "did this file, as a unit, arrive on time?" Accuracy sits in between: it's measured row by row, but it needs an external baseline — a reference price, a historical range — that no type or range rule can derive on its own.

Diagram: where each dimension lives

flowchart TB
    subgraph FILA["Row level -- one row at a time"]
        A["Completeness\nis content missing?"]
        B["Uniqueness\nwas this ID already seen?"]
        C["Validity\nare the type and range\ncorrect?"]
    end

    subgraph FILA_CONTRA_OTRA["Row level, against something external"]
        D["Consistency\ndoes the reference exist\nin another table?"]
        E["Accuracy\ndoes the value make sense\nagainst a baseline?"]
    end

    subgraph ARCHIVO["File / whole table level"]
        F["Freshness\ndid it arrive on time?"]
    end

Going deeper: why "valid" and "correct" aren't synonyms

This lesson's most important distinction — and the most commonly misunderstood — is the one that separates validity from accuracy. A value can be perfectly valid — the correct type, within the allowed range — and at the same time be completely wrong. unit_price = 60.00 in this example's accuracy_row is exactly that case: 60.00 is a positive number, it's a legitimate float, it breaks no declared range like "price can't be negative." check_validity() — if applied to this row — would find absolutely nothing wrong. And yet the value is wrong: nobody at Kiosko sells an Energy Bar for sixty dollars.

This distinction isn't a technicality — it's the underlying reason the lie of the green checkmark (lesson 2) is possible in the first place. A system that only checks validity — correct type, correct range — will let through, with a perfectly clean bill of health, any value that's technically possible but practically absurd. Catching that needs something validity, by design, doesn't have: a baseline of what's normal, to compare against. Building that baseline, deterministically and without Machine Learning, is exactly the job of this guide's module 5 — but before getting there, you need to be able to name the difference precisely, which is what this lesson gave you.

Common mistakes

Treating "completeness" and "validity" as if they were the same dimension. What happens: someone sees an empty field and describes it as "an invalid value," blending the two categories. Why it happens: both sound like "the data is malformed," and the difference feels subtle. How to spot it: if your quality report doesn't distinguish between "this field has no value at all" (completeness) and "this field has a value, but of the wrong type or range" (validity), you're losing information that matters to whoever has to fix the problem at the source — they're different causes, with different fixes. How to fix it: recall the exact distinction from check_nulls_and_types() in foundations M5 (lesson 4 of that module): first you ask whether the field has content (completeness), then whether that content has the right type (validity) — two consecutive questions, not one.

Confusing "consistency" with "accuracy." What happens: someone sees the row with product_id = "P099" and describes it as "the price is wrong," or sees the row with unit_price = 60.00 and describes it as "the product doesn't exist." Why it happens: the two dimensions share something in common — both need to look beyond the row alone —, which makes them easy to mix up at first glance. How to spot it: ask yourself what table or baseline is in play. Consistency always compares against another table (does this product_id exist in the product catalog?). Accuracy always compares against a baseline of expected values (does this price resemble what this product normally costs?). How to fix it: always name the exact reference you're comparing against — if it's "another table," it's consistency; if it's "a historical range or average," it's accuracy.

Thinking freshness can be measured row by row. What happens: someone tries to write a freshness rule that applies to each individual row, as if every order had its own arrival SLA. Why it happens: the other row-level dimensions — completeness, uniqueness, validity — really are evaluated one row at a time, so it seems natural to extend that pattern to freshness. How to spot it: if your code tries to compare a single row's order_ts against "now" to decide whether that row is fresh, you're solving the wrong question — what matters is when the whole file arrived, not when each individual sale inside it happened. How to fix it: freshness is measured once per file or per run, not once per row — this guide's module 6 builds that check at exactly that level.

Exercises

Exercise 1 — Classify without running code. Without looking at this lesson's worked example, classify each of these four problems into the correct dimension: (a) a file that was due Monday and arrives Thursday; (b) an email column that's always formatted user@domain but was never checked to confirm that person actually exists; (c) two rows with the same invoice number; (d) a country_code column with the value "XX", which isn't a real ISO code for any country.

Exercise 2 — Build your own accuracy row. Using check_accuracy() from the worked example, build a row with unit_price=1.35 for P002 (whose reference is 1.20) and run the check with tolerance=0.5 (50%). Does it get flagged as a violation? Then try with unit_price=2.00. Explain the difference.

row_1_35 = {"order_id": "ORD-D7", "product_id": "P002", "unit_price": 1.35}
row_2_00 = {"order_id": "ORD-D8", "product_id": "P002", "unit_price": 2.00}

print("1.35:", check_accuracy(row_1_35, REFERENCE_PRICE_P002, 0.5))
print("2.00:", check_accuracy(row_2_00, REFERENCE_PRICE_P002, 0.5))

1.35: False
2.00: True

Exercise 3 — Argue which dimension is hardest to automate, and why. Of this lesson's six dimensions, which do you think is hardest to verify with a simple rule, without human intervention? Justify your answer in 2-3 sentences, without looking at module 5 of this guide yet.

Summary and next step

In this lesson you built the precise vocabulary for the six dimensions of data quality — completeness, uniqueness, validity, consistency, freshness, accuracy —, each with an exact definition, an executed example, and a distinct check function. You saw, with direct evidence, why "valid" and "correct" aren't synonyms — the exact distinction that makes the lie of the green checkmark possible — and why freshness is measured at the file level, never at the individual row level.

Before moving on you should be able to: name the six dimensions without help; classify a new data problem into the correct dimension; and explain, using the unit_price=60.00 example, why a value can be valid and wrong at the same time.

With the full vocabulary in hand, lesson 4 goes back to data-engineering-foundations-guide's validate_orders() — the tool you already built — and classifies it, dimension by dimension: which of the six questions it actually answers, and which it leaves completely unanswered.

Resources

• Joe Reis & Matt Housley, Fundamentals of Data Engineering (O'Reilly, 2022) — the data quality dimensions framework this lesson rests on. oreilly.com/library/view/fundamentals-of-data/9781098108298. In English.
• Python — official datetime and timedelta documentation, the foundation for this lesson's check_freshness(). docs.python.org/3/library/datetime.html. In English.
• AWS — "AWS Certified Data Engineer - Associate (DEA-C01)" exam guide, Domain 4 ("Data Security and Governance", 18% of the exam) — the certification that formally weighs several of these six dimensions. docs.aws.amazon.com/aws-certification/latest/examguides/data-engineer-associate-01.html. In English.
• This guide's DESIGN — the full map of the eight modules, each thoroughly solving one or more of these six dimensions. src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.