When thresholds are too strict or too loose

Description

tolerance=0.5 caught ORD-9509 with plenty of margin in lesson 6 — but that result doesn't prove 0.5 is, in general, the right number. This lesson runs check_price_baseline() two more times on S04's data, with two deliberately miscalibrated thresholds: one too loose, which lets the real anomaly through; and one too strict, which flags a legitimate price variation as if it were an error. Both runs are real, with evidence, not hypothetical.

Connection to the module. This lesson completes lesson 2's work — which already warned, at the conceptual level, that tolerance is accuracy's most delicate piece — with executed evidence on S04's real case. It's the last conceptual piece before lesson 8 assembles the module's closing project.

An analogy: the smoke detector, miscalibrated in two directions

A household smoke detector has, at its core, the same kind of threshold as check_price_baseline(): a concentration of particles in the air, compared against a fixed limit. Calibrated too sensitive, the detector goes off every time someone toasts bread or showers with the door closed — after the third false alarm in a week, someone in the house ends up disconnecting the battery, and the detector stops protecting against the real fire that could actually happen. Calibrated too insensitive, by contrast, the detector might not go off until the smoke is already dense and the fire well established — protection arrives, but too late to matter.

Neither extreme is "safer" than the other — both fail, each in a different way. tolerance in check_price_baseline() has exactly this same tension: too low, it generates false alerts for every small, legitimate price variation, training the team to ignore them; too high, it lets real errors through with no warning at all. This lesson measures, with real data, exactly where those two extremes begin in Kiosko's case.

Worked example: the two extremes, with real evidence

Both blocks in this section continue directly from lesson 6's catch_dollars_to_cents.py — con, reference_prices, df, and check_price_baseline() are already loaded exactly as they were left there, with no change. Just add the new code to the end of that same file.

Extreme 1 — too loose: confusing tolerance's unit

Lesson 5 already warned, in its Common mistakes section, about confusing tolerance=50 (thinking "fifty times") with the fraction tolerance=0.5 (50%) the function actually needs. It's worth confirming, with S04's real file, exactly what happens if that mistake gets made:

# tolerance_experiments.py -- extreme 1: too loose
too_loose = check_price_baseline(df, reference_prices, tolerance=50)
print(f"tolerance=50 (confusing fraction with multiple) -> anomalies.shape: {too_loose.shape}")
print(too_loose)

What to expect.

tolerance=50 (confusing fraction with multiple) -> anomalies.shape: (0, 5)
shape: (0, 5)
┌──────────┬────────────┬────────────┬─────────────────┬───────────┐
│ order_id ┆ product_id ┆ unit_price ┆ reference_price ┆ deviation │
│ ---      ┆ ---        ┆ ---        ┆ ---             ┆ ---       │
│ str      ┆ str        ┆ f64        ┆ f64             ┆ f64       │
╞══════════╪════════════╪════════════╪═════════════════╪═══════════╡
└──────────┴────────────┴────────────┴─────────────────┴───────────┘

Zero rows. ORD-9509, with deviation=49.0 (already confirmed in lesson 6), doesn't exceed a tolerance=50 threshold — because 49.0 < 50, just barely under. The dollars-to-cents bug, this entire guide's central incident, would go completely unnoticed with this misconfigured threshold — not because of any flaw in the function, but because of a unit confusion as simple as writing 50 instead of 0.5. This result is the most direct possible confirmation of why lesson 5 dedicated a whole "Common mistake" to this specific confusion: it isn't a hypothetical textbook case — it's exactly the kind of silent error that would let this guide's real incident through.

Extreme 2 — too strict: flagging a legitimate promotion as an anomaly

Now the opposite extreme. Imagine S04, that same day, had had a legitimate, minor promotion on P001 — a water bottle sold at 0.60 instead of the usual 0.55, a completely normal price variation, the kind any business makes all the time:

# tolerance_experiments.py -- extreme 2: too strict
promo_df = df.with_columns(
    pl.when(pl.col("order_id") == "ORD-9501")
    .then(0.60)
    .otherwise(pl.col("unit_price"))
    .alias("unit_price")
)

print("--- tolerance=0.5 (this guide's default) on promo_df ---")
r1 = check_price_baseline(promo_df, reference_prices, tolerance=0.5)
print(r1.select(["order_id", "product_id", "unit_price", "reference_price", "deviation"]))

print("\n--- tolerance=0.05 (too strict) on promo_df ---")
r2 = check_price_baseline(promo_df, reference_prices, tolerance=0.05)
print(r2.select(["order_id", "product_id", "unit_price", "reference_price", "deviation"]))

pl.when(...).then(...).otherwise(...) is Polars's way of expressing a condition inside an expression — the vectorized equivalent of a Python if/else, applied column by column —: it replaces unit_price with 0.60 only in ORD-9501's row, leaving the other rows untouched.

What to expect.

--- tolerance=0.5 (this guide's default) on promo_df ---
shape: (1, 5)
┌──────────┬────────────┬────────────┬─────────────────┬───────────┐
│ order_id ┆ product_id ┆ unit_price ┆ reference_price ┆ deviation │
│ ---      ┆ ---        ┆ ---        ┆ ---             ┆ ---       │
│ str      ┆ str        ┆ f64        ┆ f64             ┆ f64       │
╞══════════╪════════════╪════════════╪═════════════════╪═══════════╡
│ ORD-9509 ┆ P002       ┆ 60.0       ┆ 1.2             ┆ 49.0      │
└──────────┴────────────┴────────────┴─────────────────┴───────────┘

--- tolerance=0.05 (too strict) on promo_df ---
shape: (2, 5)
┌──────────┬────────────┬────────────┬─────────────────┬───────────┐
│ order_id ┆ product_id ┆ unit_price ┆ reference_price ┆ deviation │
│ ---      ┆ ---        ┆ ---        ┆ ---             ┆ ---       │
│ str      ┆ str        ┆ f64        ┆ f64             ┆ f64       │
╞══════════╪════════════╪════════════╪═════════════════╪═══════════╡
│ ORD-9501 ┆ P001       ┆ 0.6        ┆ 0.55            ┆ 0.090909  │
│ ORD-9509 ┆ P002       ┆ 60.0       ┆ 1.2             ┆ 49.0      │
└──────────┴────────────┴────────────┴─────────────────┴───────────┘

With tolerance=0.5, the threshold configured across this entire guide, the result is the same one you already know: only ORD-9509. ORD-9501's legitimate promotion — a deviation of barely 9.09% — gets correctly ignored, exactly what should happen with a normal price variation. But with tolerance=0.05 (5%, ten times stricter), ORD-9501 also gets flagged, alongside ORD-9509 — a false positive: a perfectly legitimate row, treated as if it were an error, only because the configured threshold left no margin at all for normal business variation.

Diagram: the three thresholds, the same file, three different results

flowchart TD
    A["orders_s04 (with a legitimate\npromotion on ORD-9501: 0.60 instead of 0.55)"] --> B{"tolerance=50\n(unit confusion)"}
    A --> C{"tolerance=0.5\n(this guide's default)"}
    A --> D{"tolerance=0.05\n(too strict)"}

    B --> B1["0 rows flagged\nORD-9509 SLIPS THROUGH\n(false negative)"]
    C --> C1["1 row flagged\nonly ORD-9509\n(correct)"]
    D --> D1["2 rows flagged\nORD-9501 + ORD-9509\nORD-9501 is a FALSE POSITIVE"]

Going deeper: why 0.5 isn't a magic number, it's a documented decision

It's worth being honest about something this lesson demonstrates without saying it explicitly yet: tolerance=0.5 isn't the only possible "correct" value for Kiosko — it's a decision, made with judgment, that in this specific case separates, with wide margin, normal variation (9.09% in the promotion example) from a real anomaly (4900%). That wide separation — almost three orders of magnitude between the highest normal case and the lowest anomalous case this lesson built — is, in a sense, a favorable case: any reasonable threshold between, say, 15% and 1000% would have correctly distinguished both cases in this specific example. In a real business, with more frequent price variations closer to the limit — seasonal discounts of 20%, inflation adjustments of 8% — finding the right threshold would demand much more care, probably calibrated with real historical data on how much Kiosko's legitimate prices vary from one week to another, not chosen once and forgotten.

This connects directly to lesson 2's warning: tolerance is a business decision, not a technical constant. Documenting it — writing down, somewhere visible, why 0.5 was chosen and not another value — is as important as the number itself, because it's the only way someone, months later, can review whether that decision is still correct as Kiosko's business changes.

Common mistakes

Choosing an extremely low tolerance "to be safe." What happens: someone, reasoning that "a stricter threshold catches more problems," configures tolerance=0.01 (1%) expecting the maximum possible protection. Why it happens: the intuition that "stricter is safer" is reasonable in many security contexts, but not in anomaly detection with an imperfect baseline. How to spot it: if your tolerance configuration generates alerts for price variations you know, with certainty, are legitimate (rounding, minor promotions, cent-level differences between stores), your threshold is too strict — this lesson's "Extreme 2" is exactly that scenario, with evidence. How to fix it: an overly strict threshold isn't safer — it generates alert fatigue, the exact same problem this lesson's smoke detector analogy describes: after enough false alarms, someone on the team ends up ignoring the whole system, real alerts included.

Choosing a high tolerance "to avoid false positives," with no check that it still catches the real case. What happens: someone, after seeing "Extreme 2"'s false-positive problem, overcorrects toward an overly generous threshold, with no re-test against ORD-9509's known incident. Why it happens: avoiding one kind of error (false positives) is a visible, easy goal to chase without realizing you're introducing the opposite error (false negatives). How to spot it: whenever you change tolerance, run check_price_baseline() again against ORD-9509's known case — exactly what this lesson's "Extreme 1" did — and confirm it still shows up in the result. How to fix it: any threshold change needs testing against both extremes — does it still catch what it should catch? does it still let through what it should let through? — never against just one. This is, in fact, the same testing discipline module 3 already established with validate_referential_integrity(): a toy exercise with a positive case and a negative case, not just one of the two.

Thinking there's a single "correct" tolerance that works forever. What happens: someone, after finding a tolerance value that works well for S04's current file, treats it as a permanent constant that will never need review. Why it happens: once calibrated and tested, a number feels like a closed decision. How to spot it: if your tolerance configuration has no comment or documentation about when it was chosen or why, and nobody on the team knows the last time it was reviewed, you run the risk of that number stopping to reflect the business's reality with nobody noticing. How to fix it: treat tolerance, just like reference_prices (already discussed in lesson 4's exercise 3), as a decision with an expiration date — periodically reviewing whether it's still the right number, as Kiosko's catalog and price patterns evolve, is part of a data quality system's ongoing work, not a one-time task.

Exercises

Exercise 1 — Find the exact threshold where ORD-9501 (the promotion) stops getting flagged. Using this lesson's promo_df, test with tolerance=0.09 and tolerance=0.1. At which of the two does ORD-9501 stop appearing in the result?

for t in [0.09, 0.1]:
    result = check_price_baseline(promo_df, reference_prices, tolerance=t)
    print(f"tolerance={t}: {result['order_id'].to_list()}")

tolerance=0.09: ['ORD-9501', 'ORD-9509']
tolerance=0.1: ['ORD-9509']

Exercise 2 — Confirm that tolerance=0 (any deviation, however tiny, gets flagged) breaks the function in practice. Run check_price_baseline(df, reference_prices, tolerance=0) on the original orders_s04 (without ORD-9501's promotion). Which rows get flagged, and why does this confirm tolerance=0 isn't a reasonable value?

zero_tolerance = check_price_baseline(df, reference_prices, tolerance=0)
print(zero_tolerance.select(["order_id", "product_id", "unit_price", "deviation"]))

Exercise 3 — Argue why this lesson's "Extreme 1" (tolerance=50) is, in practice, more dangerous than "Extreme 2" (tolerance=0.05) for a business like Kiosko. In 3-4 sentences, compare the cost of a false negative (letting a real error through) against the cost of a false positive (flagging something normal as an error), in the specific context of the dollars-to-cents bug.

Summary and next step

In this lesson you tested check_price_baseline() at its two extremes, with real evidence: tolerance=50 (confusing fraction with multiple) lets ORD-9509 through completely unnoticed — this guide's central incident, invisible because of a single decimal digit's error —; tolerance=0.05 flags a legitimate promotion (ORD-9501 at 0.60) as if it were an error, a false positive. You confirmed that tolerance=0.5, the value this entire guide used, separates both cases with wide margin in this specific example, and you understood why that calibration is a documented business decision, not a technical constant fixed forever.

Before moving on you should be able to: explain, with the exact deviation number, why tolerance=50 lets ORD-9509 through; reproduce ORD-9501's false positive with tolerance=0.05; and argue why a false negative is, in this context, more dangerous than a false positive, without that justifying choosing an arbitrarily loose threshold.

With the seven earlier lessons complete — the diagnosis, the baseline, the function, the real result, and now its calibration — lesson 8 assembles everything into the module's closing project: S04's complete accuracy audit.

Resources

• Polars — API reference, pl.when().then().otherwise() (the vectorized conditional expression used in this lesson's "Extreme 2"). docs.pola.rs/api/python/stable/reference/functions/index.html. In English.
• Module 5, lesson 5, of this same guide ("Threshold-based anomaly detection, without Machine Learning") — the source of the original warning about confusing fraction with multiple in tolerance. src/guides/data-reliability-and-governance-guide/workbook/module-05-accuracy-and-deterministic-anomaly-detection/en/05-threshold-based-anomaly-detection-without-ml.md. In English.
• This guide's DESIGN — this lesson's mandate: exploring what happens when thresholds are miscalibrated. src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.