Module 6: Freshness Volume And Lineage

Volume checks: too few or too many rows

Next →

Description

This lesson builds this module's second file-level piece: check_volume(), a function that compares a DataFrame's row count against an expected range, and fails whether there are too few or too many. Unlike lesson 4, this time the result on S04 is going to be PASS — a deliberate contrast, to make clear this guide's quality apparatus isn't designed to find problems where none exist.

Connection to the module. Freshness (lesson 4) asked "when?". This lesson asks something different, but of the same structural nature — a property of the whole file, never of a row: "how many?". The two functions share a shape (they take a DataFrame, return a dict with a verdict), but they check completely independent dimensions of the same incident.

An analogy: a delivery truck's manifest

A truck delivering merchandise to Kiosko's stores carries, every morning, a manifest: how many boxes it's loading, destined for which store. If today's manifest says "2 boxes for S04," and that truck normally carries between 8 and 15, something went wrong before the truck left the warehouse — a loading error, a system that stopped halfway, an order that never finished. If the manifest says "40 boxes for S04," that's equally suspicious, though in the opposite direction — a full retransmission of yesterday's order, mistakenly added to today's, or two different trucks loading the same order with nobody coordinating it.

No guard checking box by box — is it properly sealed? does it have the right label? — can detect either problem, because both are failures of the total count, not of any individual box. check_volume() is, precisely, that manifest check: it doesn't care what each row contains — the other five tools in this guide already check that — it only cares whether the total row count falls within a range Kiosko has already declared reasonable.

Worked example: build, test clean, run on S04

Step 1 — the function

# checks.py -- continuation of lesson 4's file
def check_volume(df: pl.DataFrame, min_rows: int, max_rows: int) -> dict:
    """Confirms that df.height falls within [min_rows, max_rows]."""
    row_count = df.height
    return {
        "check": "volume",
        "row_count": row_count,
        "min_rows": min_rows,
        "max_rows": max_rows,
        "status": "PASS" if min_rows <= row_count <= max_rows else "FAIL",
    }

The shortest function in this entire guide so far, and on purpose: df.height — any Polars DataFrame's row count, already used with no comment since module 2 — is all it needs to answer the question. The condition min_rows <= row_count <= max_rows is a Python chained comparison, equivalent to min_rows <= row_count and row_count <= max_rows, with both limits inclusive: a file with exactly min_rows rows, or exactly max_rows, passes with no problem — the same inclusive-limits convention check_freshness() already used in lesson 4 (<=, not <).

Step 2 — test it with toy data, three scenarios

# checks.py -- continuation
if __name__ == "__main__":
    toy_ok = pl.DataFrame({"order_id": [f"T{i}" for i in range(10)]})
    toy_too_few = pl.DataFrame({"order_id": [f"T{i}" for i in range(2)]})
    toy_too_many = pl.DataFrame({"order_id": [f"T{i}" for i in range(30)]})

    print("=== check_volume on toy data ===")
    print(f"10 rows (within range):    {check_volume(toy_ok, min_rows=5, max_rows=20)}")
    print(f"2 rows (too few):          {check_volume(toy_too_few, min_rows=5, max_rows=20)}")
    print(f"30 rows (too many):        {check_volume(toy_too_many, min_rows=5, max_rows=20)}")

What to expect.

=== check_volume on toy data ===
10 rows (within range):    {'check': 'volume', 'row_count': 10, 'min_rows': 5, 'max_rows': 20, 'status': 'PASS'}
2 rows (too few):          {'check': 'volume', 'row_count': 2, 'min_rows': 5, 'max_rows': 20, 'status': 'FAIL'}
30 rows (too many):        {'check': 'volume', 'row_count': 30, 'min_rows': 5, 'max_rows': 20, 'status': 'FAIL'}

Three scenarios, three correct verdicts: 10 sits comfortably within [5, 20]; 2 is too few; 30 is too many. check_volume() already demonstrated it catches both extremes, exactly as its name promises — a two-directional check, unlike check_freshness(), which only has one limit (a file is never a problem for being reviewed "too fast").

Step 3 — run for real on orders_2026-08-14.csv

# checks.py -- continuation
    con = duckdb.connect("kiosko.duckdb")
    df = con.sql("SELECT * FROM orders_s04").pl()

    volume_result = check_volume(df, min_rows=5, max_rows=20)
    print("\n=== check_volume(df, min_rows=5, max_rows=20) ===")
    for k, v in volume_result.items():
        print(f"  {k}: {v}")

What to expect.

=== check_volume(df, min_rows=5, max_rows=20) ===
  check: volume
  row_count: 12
  min_rows: 5
  max_rows: 20
  status: PASS

row_count: 12, within [5, 20], status: PASS. This is the deliberate contrast this module's lesson 1 anticipated: not everything happening to S04 is a failure. The file has a reasonable size for a new store's first day of sales — neither suspiciously truncated, nor suspiciously inflated — and check_volume() confirms it with no warning at all. A data quality system that flagged everything as suspicious, with no distinction, would be just as untrustworthy as one that never flags anything — module 5, lesson 5 already made a similar argument about miscalibrated thresholds.

Both extremes, built on S04's real file

It's worth confirming, using S04's own file as the base — not invented toy data — what would have happened if the real file had arrived truncated or duplicated:

# volume_edge_cases.py
import duckdb
import polars as pl
from checks import check_volume

con = duckdb.connect("kiosko.duckdb")
df = con.sql("SELECT * FROM orders_s04").pl()

truncated = df.head(3)
print(f"If the file had been cut to 3 rows: {check_volume(truncated, min_rows=5, max_rows=20)}")

doubled = pl.concat([df, df])
print(f"If the whole file had been retransmitted (24 rows): {check_volume(doubled, min_rows=5, max_rows=20)}")

What to expect.

If the file had been cut to 3 rows: {'check': 'volume', 'row_count': 3, 'min_rows': 5, 'max_rows': 20, 'status': 'FAIL'}
If the whole file had been retransmitted (24 rows): {'check': 'volume', 'row_count': 24, 'min_rows': 5, 'max_rows': 20, 'status': 'FAIL'}

df.head(3) simulates the "2 boxes" truck scenario from the analogy: a file cut in half, perhaps because the process generating it failed before finishing. pl.concat([df, df]) simulates the opposite scenario: the whole file, mistakenly retransmitted, now with 24 rows instead of 12 — notice this differs from ORD-9502's duplicate module 2 already caught: that was one row repeated inside an otherwise normal file; this is the entire file duplicated, a volume problem, not a uniqueness one. Neither scenario actually happened to S04 — the real file has twelve rows, neither truncated nor duplicated — but building them with evidence, on the same real data, confirms check_volume() would catch them if they did occur.

Diagram: two file-level questions, neither about a row's content

flowchart TD
    A["orders_2026-08-14.csv (12 rows)"] --> B["check_freshness()\nWHEN the most recent data arrived"]
    A --> C["check_volume()\nHOW MANY rows the file brings"]
    B --> D["FAIL: 47.58h > 24h SLA"]
    C --> E["PASS: 12 rows within [5, 20]"]
    D --> F["A file can fail freshness\nAND pass volume\nat the same time -- they're\nindependent questions"]
    E --> F

Going deeper: where min_rows=5 and max_rows=20 come from

These two numbers weren't invented for this lesson. They already appeared, exact, in module 4: orders_contract.yaml declared sla.row_count.min: 5 and sla.row_count.max: 20 — the same range this lesson uses for check_volume(). This isn't a coincidence — it's module 4's central promise fulfilled: a data contract declares business rules in a single versioned place, and any runnable function that needs them (like check_volume(), here) reads them from there, instead of inventing them again or copying them from memory. The business reasoning behind those numbers — fewer than 5 suggests a truncated file, more than 20 suggests a full retransmission, for a new store's first day of sales — was already precisely explained by module 4, lesson 3 when writing the contract. This lesson doesn't repeat that reasoning: it executes it.

Common mistakes

Thinking check_volume() and check_freshness() measure the same thing, because both are "whole-file." What happens: someone, after seeing the two functions share a shape (they take df, return a dict with status), assumes a file that fails one necessarily fails the other, or that running just one of the two is enough. Why it happens: the structural similarity (both file-level, both with a binary verdict) invites thinking they're also similar in content. How to spot it: review S04's real result in this guide — check_freshness() gave FAIL, check_volume() gave PASS, on the same file, at the same time. How to fix it: treat each file-level check as completely independent from the others — this lesson's diagram shows it precisely: a file can fail freshness and pass volume simultaneously, because they measure different business questions (when versus how many), with no logical relationship between them.

Using len(df) instead of df.height and assuming they're always interchangeable. What happens: someone writes len(df) instead of df.height, familiar with the standard Python pattern (len() on lists, dictionaries, text strings). Why it happens: Polars does support len(df) as a valid alias for df.height — so this "mistake" doesn't actually break anything in practice, but it's worth knowing why this lesson explicitly prefers df.height. How to spot it: if your code mixes len(df) in some places and df.height in others within the same project, with no criterion at all, your style is inconsistent even if it works. How to fix it: this guide consistently prefers df.height because it's explicit about what it's measuring — a two-dimensional DataFrame's row dimension — while len() is a generic Python function whose meaning depends on the type of object it receives. It isn't a functional error, it's a clarity preference this guide maintains in every lesson.

Choosing a max_rows with no margin at all, glued to the "normal" day's row count. What happens: someone, configuring check_volume() for a new store, sets max_rows exactly equal to the first real file's row count they saw (say, max_rows=12 for S04, instead of 20), leaving no margin for the business's normal growth. Why it happens: it seems "more precise" to tighten the limit to the already-known data. How to spot it: if your max_rows is identical to the first file's row_count you saw, any normal growth — a new store gaining customers, a day with more traffic than usual — would trigger a volume FAIL with no real data problem at all. How to fix it: module 4's contract left a deliberate margin — max_rows=20 against 12 real rows, almost double — precisely to absorb normal business variation without constantly generating false alerts. A useful volume limit sits far enough from the typical case to not trigger on every normal fluctuation, but close enough to still catch something genuinely anomalous.

Exercises

Exercise 1 — Confirm the two exact limits, 5 and 20, as edge cases. Using df.head(5) and a concatenation that produces exactly 20 rows, confirm both cases give PASS (the limits are inclusive).

See solution 
five = df.head(5)
print(f"Exactly 5 rows: {check_volume(five, min_rows=5, max_rows=20)}")

twenty = pl.concat([df, df.head(8)])
print(f"Exactly 20 rows: {check_volume(twenty, min_rows=5, max_rows=20)}")

Expected output:

Exactly 5 rows: {'check': 'volume', 'row_count': 5, 'min_rows': 5, 'max_rows': 20, 'status': 'PASS'}
Exactly 20 rows: {'check': 'volume', 'row_count': 20, 'min_rows': 5, 'max_rows': 20, 'status': 'PASS'}

Both limits are inclusive, confirmed with evidence — min_rows <= row_count <= max_rows accepts both exact extremes as valid, the same convention check_freshness() already used with <= in lesson 4.

Exercise 2 — Confirm what happens with a completely empty DataFrame. Build a DataFrame with 0 rows and run check_volume() on it with min_rows=5, max_rows=20. Is it a special case, or does the same code already handle it correctly?

See solution 
empty_df = pl.DataFrame({"order_id": []}, schema={"order_id": pl.Utf8})
print(check_volume(empty_df, min_rows=5, max_rows=20))

Expected output:

{'check': 'volume', 'row_count': 0, 'min_rows': 5, 'max_rows': 20, 'status': 'FAIL'}

FAIL, with no special handling needed — an empty DataFrame's df.height is simply 0, and 5 <= 0 <= 20 is naturally False, exactly like any other below-minimum case. This confirms check_volume() needs no special case (if df.height == 0: ...) to handle a completely empty file — a file with zero rows already falls, with no additional work, inside the same logic that catches any "too small" file.

Exercise 3 — Argue why ORD-9502's duplicate (module 2) and this lesson's fully duplicated file are related failures, but caught by different tools. In 2-3 sentences, explain the granularity difference between the two problems, and why neither tool — OrdersSchema (uniqueness) or check_volume() — could fully replace the other.

See solution

Both problems share a plausible root cause — a retransmission, something resent by mistake —, but they occur at different scales: ORD-9502 is a single row repeated inside an otherwise normal file (11 distinct order_id among 12 rows), while this lesson's scenario duplicates the entire file, all twelve rows repeated exactly. OrdersSchema (with unique=True on order_id) would catch the first case but not the second directly — a 24-row file where every order_id repeats exactly once would still have duplicate pairs detectable by uniqueness, but the most immediate, easy-to-detect symptom in that scenario is simply that the file has double the expected rows, the question check_volume() answers much more directly and quickly than checking duplicates one by one.

Summary and next step

In this lesson you built check_volume(), this module's second file-level function. You tested it on toy data with three scenarios (within range, too few, too many rows), confirmed with evidence that both limits are inclusive, and ran it on real S04: row_count: 12, status: PASS — the deliberate contrast with the earlier lesson's freshness FAIL. You also built, on S04's real file (not invented data), the two failure scenarios check_volume() would catch: a file truncated to 3 rows, and one fully duplicated to 24.

Before moving on you should be able to: explain why check_freshness() and check_volume() are completely independent from each other, with S04's real result as evidence; and explain where the 5 and 20 numbers come from, with no need to invent them again.

With freshness and volume resolved, this guide's file-level diagnosis is complete. Lessons 6 and 7 change topic entirely: from "is it okay?" to "where does it come from?" — the lineage question, which none of the seven tools built so far can answer.

Resources

  • Polars — official documentation, DataFrame.height (the property used in this lesson to count rows). docs.pola.rs. In English.
  • Module 4, lesson 3, of this same guide ("Writing orders_contract.yaml") — the exact source of sla.row_count.min: 5 and sla.row_count.max: 20, the numbers this lesson reuses. src/guides/data-reliability-and-governance-guide/workbook/module-04-data-contracts-as-versioned-artifacts/en/03-writing-orders-contract-yaml.md. In English.
  • Module 2, lesson 6, of this same guide — the source of ORD-9502's duplicate contrasted in this lesson's Exercise 3. src/guides/data-reliability-and-governance-guide/workbook/module-02-declarative-data-quality-tests-with-pandera/en/06-completeness-and-uniqueness-checks.md. In English.
  • This guide's DESIGN — check_volume(df, min_rows=5, max_rows=20)'s exact mandate. src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.
← PreviousWriting a freshness check for S04Next →What lineage answers that a contract does not