Threshold-based anomaly detection, without Machine Learning

Description

This lesson writes check_price_baseline() — the function that compares every row of a DataFrame against reference_prices, and decides, with a transparent, single-line formula, whether the deviation is large enough to flag. Before touching S04's real data — that's lesson 6 — you test it on a small, controlled toy example, following the same pedagogical order module 3 already used with validate_referential_integrity().

Connection to the module. This lesson brings together the two pieces lessons 2 through 4 left prepared: the relative deviation formula check_accuracy() already used in module 1, and reference_prices, built in lesson 4. The result is the complete tool, still not applied to the real incident — that arrives in lesson 6.

An analogy: the security guard who doesn't need to recognize faces

A security guard at an office building's entrance doesn't need to recognize each of the thousand people who work there. A much simpler rule is enough: every person entering has to swipe their badge through a reader, and the reader compares that badge against a list of active badges. If the badge is on the list, they pass; if it isn't, they don't. The guard needs no sophisticated facial recognition model trained on thousands of photos — they need a deterministic rule, a reference list, and a binary comparison criterion.

check_price_baseline() is that same guard, applied to prices instead of people. It doesn't need to "learn" what price looks suspicious from thousands of historical examples — that would be, precisely, Machine Learning's territory, which this guide excludes. It needs a reference list (reference_prices, already built in lesson 4) and an explicit comparison rule: does the difference between this row's price and its product's reference price exceed a fixed threshold? Yes or no, with no middle ground, with no trained model involved.

Worked example: check_price_baseline(), first on toy data

Step 1 — the complete function

# price_baseline_toy.py
import polars as pl


def check_price_baseline(
    df: pl.DataFrame, reference_prices: dict[str, float], tolerance: float = 0.5
) -> pl.DataFrame:
    """Rows in df whose unit_price deviates from the baseline beyond `tolerance`
    (a fraction, not a percentage -- 0.5 means 50%).

    Rows with no unit_price (completeness) or with a product_id outside
    reference_prices (consistency) have no baseline to compare against -- they
    get excluded here, not because they're fine, but because this function
    isn't the tool that catches them.
    """
    return (
        df.with_columns(
            pl.col("product_id").replace_strict(reference_prices, default=None).alias("reference_price")
        )
        .filter(pl.col("unit_price").is_not_null() & pl.col("reference_price").is_not_null())
        .with_columns(
            ((pl.col("unit_price") - pl.col("reference_price")).abs() / pl.col("reference_price"))
            .alias("deviation")
        )
        .filter(pl.col("deviation") > tolerance)
    )

Read the function in four steps, in the same order Polars executes them. First, pl.col("product_id").replace_strict(reference_prices, default=None) — Polars's Expr.replace_strict() method substitutes every product_id value with its matching entry in the reference_prices dictionary; the default=None parameter tells it that, for any product_id not in the dictionary (like P099), the result should be null instead of raising an error. Second, .filter() discards rows with no unit_price (completeness, already covered by another tool) or with no known reference_price (product_id outside the catalog, already covered by module 3's validate_referential_integrity()) — this function doesn't try to cover the others' work. Third, it calculates deviation: the exact same formula check_accuracy() already used in module 1, lesson 3 — abs(value - reference) / reference —, now as a vectorized Polars expression applied to the remaining rows all at once, with no explicit Python loop. Fourth, the final .filter() keeps only rows whose deviation exceeds tolerance — the same return contract validate_orders(), SchemaErrors.failure_cases, and validate_referential_integrity() already followed: the function returns exclusively the rows that fail, never the ones that pass.

Step 2 — toy data, with one clearly anomalous case and one clearly normal one

# price_baseline_toy.py -- continuation
toy_orders = pl.DataFrame({
    "order_id": ["ORD-T1", "ORD-T2", "ORD-T3"],
    "product_id": ["P001", "P001", "P002"],
    "unit_price": [0.55, 0.58, 45.00],
})
toy_reference_prices = {"P001": 0.55, "P002": 1.20}

print("toy_orders:")
print(toy_orders)

anomalies = check_price_baseline(toy_orders, toy_reference_prices, tolerance=0.5)
print(f"\ncheck_price_baseline(toy_orders, toy_reference_prices, tolerance=0.5):")
print(anomalies)

What to expect.

toy_orders:
shape: (3, 3)
┌──────────┬────────────┬────────────┐
│ order_id ┆ product_id ┆ unit_price │
│ ---      ┆ ---        ┆ ---        │
│ str      ┆ str        ┆ f64        │
╞══════════╪════════════╪════════════╡
│ ORD-T1   ┆ P001       ┆ 0.55       │
│ ORD-T2   ┆ P001       ┆ 0.58       │
│ ORD-T3   ┆ P002       ┆ 45.0       │
└──────────┴────────────┴────────────┘

check_price_baseline(toy_orders, toy_reference_prices, tolerance=0.5):
shape: (1, 5)
┌──────────┬────────────┬────────────┬─────────────────┬───────────┐
│ order_id ┆ product_id ┆ unit_price ┆ reference_price ┆ deviation │
│ ---      ┆ ---        ┆ ---        ┆ ---             ┆ ---       │
│ str      ┆ str        ┆ f64        ┆ f64             ┆ f64       │
╞══════════╪════════════╪════════════╪═════════════════╪═══════════╡
│ ORD-T3   ┆ P002       ┆ 45.0       ┆ 1.2             ┆ 36.5      │
└──────────┴────────────┴────────────┴─────────────────┴───────────┘

Exactly one anomalous row, ORD-T3, with deviation=36.5 (a 3650% deviation over the reference price). ORD-T1 (0.55, identical to the reference price) has deviation=0.0 — well below tolerance=0.5, it doesn't get flagged. ORD-T2 (0.58, a small 5.45% variation over 0.55) doesn't get flagged either — a difference of a few cents, the kind a legitimate promotion or a different rounding could explain without it being a real error. The result confirms, with controlled data where you already knew beforehand which row should fail, that the function correctly distinguishes between normal variation and a real anomaly.

Diagram: check_price_baseline()'s complete mechanics

flowchart TD
    A["df: rows with\nproduct_id, unit_price"] --> B["replace_strict(reference_prices,\ndefault=None)"]
    B --> C{"is reference_price\nnull?"}
    C -->|"Yes (product_id\noutside the catalog)"| D["excluded\n(consistency, another check)"]
    C -->|"No"| E{"is unit_price\nnull?"}
    E -->|"Yes"| D2["excluded\n(completeness, another check)"]
    E -->|"No"| F["deviation =\nabs(unit_price - reference_price)\n/ reference_price"]
    F --> G{"deviation > tolerance?"}
    G -->|"Yes"| H["KEPT\n-- anomaly"]
    G -->|"No"| I["discarded\n-- within normal"]

Going deeper: why a fraction, not a percentage or an absolute difference

It's worth being explicit about a design decision in deviation that might seem arbitrary: why divide by reference_price, instead of using the absolute difference directly? An absolute difference — abs(unit_price - reference_price) > tolerance, with no division — would have a serious problem with a price catalog as varied as Kiosko's: an absolute threshold of, say, 2.00, would flag any deviation of two dollars or more as anomalous, regardless of the product. For P001 (0.55), a 2.00 deviation would be absurd — almost four times the normal price —; for P004 (4.50), that same 2.00 absolute deviation would, by contrast, be reasonable — less than half the normal price. A single absolute threshold can't be correct for both products at once.

Dividing by reference_price normalizes the problem: this lesson's deviation always gets expressed as a fraction of that specific product's normal price, so the same tolerance=0.5 (50%) means something proportional and coherent regardless of whether the product costs fifty cents or four and a half dollars. This is, in fact, the same mathematical reason infrastructure monitoring systems almost never alert on "latency went up 100 milliseconds" in absolute terms, and almost always alert on "latency went up 300% over its baseline" — relative deviation generalizes better when the scale of normal values varies a lot from one case to another.

Common mistakes

Omitting default=None in replace_strict(), and hitting an error on the first unknown product. What happens: someone writes pl.col("product_id").replace_strict(reference_prices), with no default parameter, and the function works perfectly on toy data where every product_id is in the dictionary — until a row arrives with a product outside the catalog, like P099.

df_with_unknown = pl.DataFrame({
    "order_id": ["ORD-T1", "ORD-T2"],
    "product_id": ["P001", "P099"],
    "unit_price": [0.55, 1.00],
})
reference_prices = {"P001": 0.55, "P002": 1.20, "P003": 0.75, "P004": 4.50}

result = df_with_unknown.with_columns(
    pl.col("product_id").replace_strict(reference_prices).alias("reference_price")
)

InvalidOperationError: incomplete mapping specified for `replace_strict`

Hint: Pass a `default` value to set unmapped values.

Why it happens: replace_strict(), as Polars's official documentation confirms, requires every non-null value in the column to have an entry in the mapping — it's a deliberate protection against silently "forgetting" a value, not an API oversight. How to spot it: the error message names the exact problem ("incomplete mapping") and suggests the fix in the same message ("Pass a default value") — it's one of the few errors in this guide that explains itself. How to fix it: always pass default=None (or whatever value makes sense as "there's no baseline for this") whenever a column's possible values could include something outside your reference dictionary — exactly as check_price_baseline() does in this lesson, with full awareness that S04 is going to bring product_id="P099".

Confusing tolerance=0.5 with "50 times" instead of "50%." What happens: someone, thinking of ORD-9509's real case (a deviation of 50 times the normal price), passes tolerance=50 expecting it to be "the correct threshold to catch a 50x deviation." Why it happens: the number "50" appears in the incident's narrative, and it's easy to copy it without checking tolerance's units. How to spot it: if your call to check_price_baseline() uses a tolerance value greater than 1.0, you probably confused a fraction with a multiple — tolerance=1.0 already means "100% deviation," an extremely loose threshold. How to fix it: remember deviation is a fraction (0.5 is 50%, not 50x); this module's lesson 7 goes deep, with executed evidence, into exactly what happens if this specific mistake gets made on S04's real case.

Writing the deviation comparison with >= instead of >, and not noticing the difference in practice. What happens: someone changes pl.col("deviation") > tolerance to >= tolerance, thinking it's equivalent or even "safer." Why it happens: in most cases with real data, the difference between > and >= doesn't change any result, because it's extremely unlikely for a deviation to land exactly on the threshold. How to spot it: build a test case where deviation is exactly equal to tolerance (say, unit_price=0.825 with reference_price=0.55 and tolerance=0.5, since (0.825-0.55)/0.55 = 0.5 exactly) and compare the result with > against >=. How to fix it: this guide always uses strict > — a row exactly at the tolerance limit gets considered still "within normal," not anomalous; it's a convention, not an absolute mathematical truth, and it's worth documenting explicitly if your team decides otherwise.

Exercises

Exercise 1 — Reproduce the toy example with a different tolerance. Using this lesson's toy_orders and toy_reference_prices, run check_price_baseline() with tolerance=0.1 (10%, much stricter than the example's 0.5). How many rows get flagged now?

strict_anomalies = check_price_baseline(toy_orders, toy_reference_prices, tolerance=0.1)
print(strict_anomalies)

shape: (1, 5)
┌──────────┬────────────┬────────────┬─────────────────┬───────────┐
│ order_id ┆ product_id ┆ unit_price ┆ reference_price ┆ deviation │
│ ---      ┆ ---        ┆ ---        ┆ ---             ┆ ---       │
│ str      ┆ str        ┆ f64        ┆ f64             ┆ f64       │
╞══════════╪════════════╪════════════╪═════════════════╪═══════════╡
│ ORD-T3   ┆ P002       ┆ 45.0       ┆ 1.2             ┆ 36.5      │
└──────────┴────────────┴────────────┴─────────────────┴───────────┘

Exercise 2 — Add a row with an unknown product_id to toy_orders, and confirm it doesn't break the function. Add a fourth row to toy_orders with product_id="P999" (which doesn't exist in toy_reference_prices) and unit_price=10.00. Run check_price_baseline() again. Does that row show up in the result?

toy_orders_with_unknown = pl.concat([
    toy_orders,
    pl.DataFrame({"order_id": ["ORD-T4"], "product_id": ["P999"], "unit_price": [10.00]}),
])

result = check_price_baseline(toy_orders_with_unknown, toy_reference_prices, tolerance=0.5)
print(result)

Exercise 3 — Explain, with no code execution, why reversing the order of the .filter() and .with_columns() operations in the function would break the result. This lesson's function first filters out rows with no known reference_price, and only then calculates deviation. In 2-3 sentences, explain what would happen if deviation were calculated before that filter, for rows with an unknown product_id.

Summary and next step

In this lesson you wrote check_price_baseline(), this module's central function: four steps — mapping product to reference price with replace_strict(), filtering out rows with insufficient data, calculating relative deviation, filtering against the tolerance —, tested first on toy data where you already knew beforehand which row should get flagged. You understood, with a concrete numeric comparison, why deviation gets calculated as a fraction relative to the reference price, not as an absolute difference or an unnormalized percentage. And you confirmed with evidence, triggering the error on purpose, that replace_strict() requires default=None so it doesn't break on unknown products.

Before moving on you should be able to: explain each of check_price_baseline()'s four steps without looking at the code; explain why an absolute difference would be a poor threshold for a catalog with prices as varied as Kiosko's; and reproduce replace_strict()'s exact error with no default.

You have the function ready, tested, verified against data where you already knew the expected result. Lesson 6 runs this exact same function, with no change at all, over orders_2026-08-14.csv's twelve real rows — the moment when ORD-9509, after four complete modules, finally gets flagged.

Resources

• Polars — API reference, Expr.replace_strict() (exact behavior against unmapped values, the default parameter). docs.pola.rs/api/python/stable/reference/expressions/api/polars.Expr.replace_strict.html. In English.
• Module 1, lesson 3, of this same guide ("The six data quality dimensions") — the original source of the relative deviation formula (check_accuracy()) this lesson vectorizes with Polars. src/guides/data-reliability-and-governance-guide/workbook/module-01-when-green-does-not-mean-correct/en/03-six-dimensions-of-data-quality.md. In English.
• Module 3, lesson 4, of this same guide ("Writing a referential integrity check") — the source of the same pedagogical pattern (test on toy data before touching real S04) this lesson follows. src/guides/data-reliability-and-governance-guide/workbook/module-03-consistency-and-referential-checks/en/04-writing-a-referential-integrity-check.md. In English.
• This guide's DESIGN — check_price_baseline(df, reference_prices, tolerance=0.5)'s exact signature and the Machine Learning boundary. src/guides/data-reliability-and-governance-guide/DISENO.md. In Spanish.