Module 7: Observability Latency And Evals In Production

6. Hands-on: the smoke test harness with fixed manifests

Next →

Description

This lesson builds evals/manifest_extraction_smoke_test.py, run over evals/fixtures/sample_manifests.json — extract-shipment-manifest-fields's smoke test harness, the piece lesson 5 promised without building. The comparison structure — loading five fixed cases, validating each against ShipmentFields, producing a PASS/FAIL report — is real code, really executed to write this lesson, reusing validate_shipment_fields() (Module 4, lesson 6) without modifying a single line. The model response each case compares is a fixed, hand-written dict, explicitly labeled "REPRESENTATIVE" inside the data file itself — never a real invocation's output, for the reasons lesson 7 develops with complete precision.

Connection to the module

Lesson 5 drew the line: shape (this guide) versus meaning (AI Engineering). This lesson builds exactly on the "shape" side of that line — every assertion in this harness asks "does this response have the correct keys, with non-empty values, weightKg numeric?", never "is this the correct extraction for this text?" Lesson 2's SLI 3 (guardrail block rate) gets, here, its first real number: two of this harness's five cases fail validation, a 40% block rate over this specific set.

Analogy: an assembly line's quality control, run against sample parts

A factory that wants to test its quality-control line — does it really catch a box with a missing item? — doesn't wait for real production boxes to arrive to test it. It deliberately assembles a fixed set of test boxes: some perfect, some with an item deliberately missing, some with an extra item that shouldn't be there. It runs those test boxes through the line, and confirms quality control correctly classifies them — perfect ones as perfect, defective ones as defective. This proves quality control works, not that the factory is producing perfect boxes today. manifest_extraction_smoke_test.py is exactly that exercise: this harness's five "boxes" aren't real Bedrock extractions — they're hand-built, some deliberately complete, some deliberately incomplete — to test that validate_shipment_fields(), the quality control, classifies them correctly. The day a real account with Bedrock enabled exists, this same harness would run against real responses without changing a single line of its comparison logic.

Step 1 — The fixtures file, five cases, extending 4471/4472/4473

evals/fixtures/sample_manifests.json, at the root of andes-cargo-infra/:

[
  {
    "caseId": "4471",
    "rawText": "Hi team,\n\nFollowing up on the shipment we discussed on the call. We're sending\n120kg of textile goods from our Lima warehouse to the distribution\ncenter in Santiago. AndesExpress is handling the pickup this Thursday.\nShipment reference on our side is AC-4471.\n\nRegards,\nLogistics Team\n",
    "expectedFields": {
      "shipmentId": "4471",
      "originCountry": "Peru",
      "destinationCountry": "Chile",
      "carrier": "AndesExpress",
      "weightKg": "120"
    },
    "representativeModelResponse": {
      "shipmentId": "4471",
      "originCountry": "Peru",
      "destinationCountry": "Chile",
      "carrier": "AndesExpress",
      "weightKg": "120"
    },
    "note": "REPRESENTATIVE -- hand-built to match, never a real Bedrock response. Module 1, lesson 3 source."
  },
  {
    "caseId": "4472",
    "rawText": "Hello,\n\nWe have 85kg of goods ready to move from our Bogota facility to\nQuito. AndesExpress will handle the transport, pickup scheduled for\nnext week. Our reference for this one is CO-4472.\n\nThanks,\nWarehouse Team\n",
    "expectedFields": {
      "shipmentId": "4472",
      "originCountry": "Colombia",
      "destinationCountry": "Ecuador",
      "carrier": "AndesExpress",
      "weightKg": "85"
    },
    "representativeModelResponse": {
      "shipmentId": "4472",
      "originCountry": "Colombia",
      "destinationCountry": "Ecuador",
      "carrier": "AndesExpress",
      "weightKg": "85"
    },
    "note": "REPRESENTATIVE -- hand-built to match, never a real Bedrock response."
  },
  {
    "caseId": "4473",
    "rawText": "Team,\n\nPlease process this shipment: 200kg, moving from Santiago back to\nLima this time, carrier is RutaSur as usual. Shipment ref CL-4473.\n\nRegards,\nOps\n",
    "expectedFields": {
      "shipmentId": "4473",
      "originCountry": "Chile",
      "destinationCountry": "Peru",
      "carrier": "RutaSur",
      "weightKg": "200"
    },
    "representativeModelResponse": {
      "shipmentId": "4473",
      "originCountry": "Chile",
      "destinationCountry": "Peru",
      "carrier": "RutaSur"
    },
    "note": "REPRESENTATIVE -- deliberately incomplete (missing weightKg), standing in for a plausible extraction gap on a genuinely ambiguous input."
  },
  {
    "caseId": "4475",
    "rawText": "Good morning,\n\nWe have a new shipment ready for pickup: 60kg of electronics parts,\norigin Quito, destination Lima, carried by RutaSur. Please confirm\nonce it is scheduled. Our internal reference is EQ-4475.\n\nBest,\nPartner Logistics Desk\n",
    "expectedFields": {
      "shipmentId": "4475",
      "originCountry": "Ecuador",
      "destinationCountry": "Peru",
      "carrier": "RutaSur",
      "weightKg": "60"
    },
    "representativeModelResponse": {
      "shipmentId": "4475",
      "originCountry": "Ecuador",
      "destinationCountry": "Peru",
      "carrier": "RutaSur",
      "weightKg": "60"
    },
    "note": "REPRESENTATIVE -- hand-built to match, never a real Bedrock response. Extends the fixed batch beyond 4471/4472/4473."
  },
  {
    "caseId": "4476",
    "rawText": "Hi,\n\nOne more for this week: 45kg general cargo, from Cali to Guayaquil,\nAndesExpress again. Reference AC-4476.\n\nThanks,\nLogistics Team\n",
    "expectedFields": {
      "shipmentId": "4476",
      "originCountry": "Colombia",
      "destinationCountry": "Ecuador",
      "carrier": "AndesExpress",
      "weightKg": "45"
    },
    "representativeModelResponse": {
      "shipmentId": "4476",
      "originCountry": "Colombia",
      "destinationCountry": "Ecuador",
      "carrier": "AndesExpress",
      "weightKg": "45",
      "confidenceScore": "high"
    },
    "note": "REPRESENTATIVE -- deliberately adds an invented field (confidenceScore) never requested, standing in for a plausible over-generation failure mode."
  }
]

Five cases: the three already-known shipments (4471, 4472, 4473) plus two new ones (4475, 4476), exactly as this module's design promised. Notice each one's note field: the word "REPRESENTATIVE" appears in all five, with no exception — the label doesn't live in this lesson's prose, it lives inside the data file itself, so no one who opens sample_manifests.json in the future, without having read this lesson, can confuse representativeModelResponse with a real response. And notice cases 4473 and 4476: one is missing a field, the other has an extra one — deliberately, so the harness has something real to reject, not just cases that always pass.

Step 2 — The harness, complete

evals/manifest_extraction_smoke_test.py:

#!/usr/bin/env python3
"""manifest_extraction_smoke_test.py -- the structural smoke test harness for
extract-shipment-manifest-fields (Module 7, lesson 6).

What this script IS: a real, executed comparison of SHAPE -- does a candidate
response have the five ShipmentFields keys, non-empty, weightKg numeric, no
extra keys? It reuses validate_shipment_fields() from
guardrails/post_invoke_checks.py (Module 4, lesson 6) verbatim, unmodified --
the same schema check that already guards the real write path to Shipments.

What this script IS NOT: a semantic quality eval. It never asks "did the
model understand this manifest correctly?" -- only "does the candidate have
the right shape?". Module 7, lesson 5 draws this line in prose; this script
draws it in code. Semantic quality evaluation is AI Engineering's job (Module
1, lesson 4), not this guide's.

Every "representativeModelResponse" in evals/fixtures/sample_manifests.json is
a hand-built dict, labeled as such in its own "note" field -- never the output
of a real Bedrock invocation (Module 7, lesson 7 explains exactly why none was
invoked). The comparison logic below is 100% real and executed; the data it
compares is representative.

Never uses random or datetime.now(). Same fixtures file, same result, every
run. Run with:
    python3 evals/manifest_extraction_smoke_test.py
    pytest evals/test_manifest_extraction_smoke_test.py -v
"""

from __future__ import annotations

import json
import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "guardrails"))

from post_invoke_checks import validate_shipment_fields  # noqa: E402

FIXTURES_PATH = Path(__file__).resolve().parent / "fixtures" / "sample_manifests.json"


def load_fixtures(path: Path = FIXTURES_PATH) -> list[dict]:
    return json.loads(path.read_text())


def run_smoke_test(fixtures: list[dict]) -> list[dict]:
    """For each fixture, validate representativeModelResponse against
    ShipmentFields. Returns one result dict per fixture -- never compares
    against expectedFields directly (that would be a semantic check: "is
    this the RIGHT extraction for this text?", out of scope here). This
    harness only asks: "is the candidate's SHAPE writable to Shipments?"."""
    results = []
    for fixture in fixtures:
        candidate = fixture["representativeModelResponse"]
        validation = validate_shipment_fields(candidate)
        results.append(
            {
                "caseId": fixture["caseId"],
                "isValid": validation.is_valid,
                "errors": validation.errors,
            }
        )
    return results


def format_summary(results: list[dict]) -> str:
    lines = []
    passed = sum(1 for r in results if r["isValid"])
    total = len(results)
    for r in results:
        status = "PASS" if r["isValid"] else "FAIL"
        lines.append(f"[{r['caseId']}] {status}")
        for err in r["errors"]:
            lines.append(f"        - {err}")
    lines.append("")
    lines.append(f"Smoke test: {passed}/{total} fixtures passed structural validation")
    return "\n".join(lines)


def main() -> int:
    fixtures = load_fixtures()
    results = run_smoke_test(fixtures)
    print(format_summary(results))
    return 0 if all(r["isValid"] for r in results) else 1


if __name__ == "__main__":
    raise SystemExit(main())

Notice run_smoke_test()'s comment: it explicitly says it never compares against expectedFields. It's a deliberate design decision, not an oversight — comparing representativeModelResponse against expectedFields field by field would be, precisely, the first step toward semantic evaluation ("does the response match what should have been extracted?"), the line lesson 5 already carefully drew. This harness validates against ShipmentFields — the schema, the shape — never against expectedFields — the correct content. The expectedFields field exists in the fixtures file as documentation of what extraction would be correct, useful for a human reader, but this harness's code never reads it.

Step 3 — Running the harness, for real

python3 evals/manifest_extraction_smoke_test.py

What to expect (literal — the structural comparison really ran; the data it compares is representative, labeled inside the file itself):

[4471] PASS
[4472] PASS
[4473] FAIL
        - missing required field(s): weightKg
[4475] PASS
[4476] FAIL
        - unexpected field(s) not in ShipmentFields: confidenceScore

Smoke test: 3/5 fixtures passed structural validation

Three out of five. Notice the two cases that fail, and that each fails for a different reason — 4473 for a missing field, 4476 for an unexpected field, the two types of failure post_invoke_checks.py already precisely distinguished in Module 4, lesson 6. No case fails for a reason invented for this lesson; both reuse, unchanged, validate_shipment_fields()'s logic, which already ran sixteen times in that module.

echo $?
1

Exit code 1 — the same mechanism a CI step would use to stop a pipeline: if this harness ever ran automatically in ci.yml against real responses, a result like this one (3/5, not 5/5) would block the merge, exactly like any other check in this ecosystem.

Step 4 — The pytest suite, six fixed cases

evals/test_manifest_extraction_smoke_test.py:

"""pytest suite for manifest_extraction_smoke_test.py -- exercises the
harness itself against the fixed fixtures file. No random, no
datetime.now(). Run with:
    pytest evals/test_manifest_extraction_smoke_test.py -v
"""

from manifest_extraction_smoke_test import load_fixtures, run_smoke_test


def test_five_fixtures_loaded():
    fixtures = load_fixtures()
    assert len(fixtures) == 5


def test_every_fixture_has_a_representative_label():
    fixtures = load_fixtures()
    for fixture in fixtures:
        assert "REPRESENTATIVE" in fixture["note"]


def test_three_of_five_fixtures_pass_structural_validation():
    fixtures = load_fixtures()
    results = run_smoke_test(fixtures)
    passed = [r for r in results if r["isValid"]]
    assert len(passed) == 3


def test_4473_fails_on_missing_weight_kg():
    fixtures = load_fixtures()
    results = run_smoke_test(fixtures)
    result_4473 = next(r for r in results if r["caseId"] == "4473")
    assert result_4473["isValid"] is False
    assert "missing required field(s): weightKg" in result_4473["errors"][0]


def test_4476_fails_on_unexpected_field():
    fixtures = load_fixtures()
    results = run_smoke_test(fixtures)
    result_4476 = next(r for r in results if r["caseId"] == "4476")
    assert result_4476["isValid"] is False
    assert "confidenceScore" in result_4476["errors"][0]


def test_4471_4472_4475_pass():
    fixtures = load_fixtures()
    results = run_smoke_test(fixtures)
    for case_id in ("4471", "4472", "4475"):
        result = next(r for r in results if r["caseId"] == case_id)
        assert result["isValid"] is True
cd evals/
pytest test_manifest_extraction_smoke_test.py -v -p no:randomly

What to expect (literal — really run):

============================= test session starts ==============================
platform darwin -- Python 3.14.0, pytest-9.1.1, pluggy-1.6.0
rootdir: evals/
collected 6 items

test_manifest_extraction_smoke_test.py::test_five_fixtures_loaded PASSED [ 16%]
test_manifest_extraction_smoke_test.py::test_every_fixture_has_a_representative_label PASSED [ 33%]
test_manifest_extraction_smoke_test.py::test_three_of_five_fixtures_pass_structural_validation PASSED [ 50%]
test_manifest_extraction_smoke_test.py::test_4473_fails_on_missing_weight_kg PASSED [ 66%]
test_manifest_extraction_smoke_test.py::test_4476_fails_on_unexpected_field PASSED [ 83%]
test_manifest_extraction_smoke_test.py::test_4471_4472_4475_pass PASSED  [100%]

============================== 6 passed in 0.01s ===============================

Six out of six, including test_every_fixture_has_a_representative_label — the test that exists, specifically, so no new case anyone adds to sample_manifests.json in the future can sneak in without the "REPRESENTATIVE" label in its note field. It's the same honesty discipline as the rest of this guide, now applied as a pytest assertion that would automatically fail if someone forgot it.

Step 5 — The guardrail block rate, with a real number (over this batch)

Lesson 2's SLI 3 — the custom guardrail block rate — gets, here, its first concrete data point:

   Block rate = FAIL candidates / evaluated candidates
              = 2 / 5
              = 40%

With the exact same honesty lesson 4's Step 5 already applied to the escalation rate: this 40% is real over these five specific candidates, deliberately built to include two types of failure — it isn't a measurement of how often a real Bedrock invocation would produce an incomplete response. What it does demonstrate, with executed evidence: validate_shipment_fields() correctly distinguishes between complete and incomplete candidates, unambiguously, one hundred percent of the time, over any candidate presented to it — the guarantee that makes this SLI trustworthy the day real traffic to evaluate actually exists.

Common mistakes

Modifying run_smoke_test() to compare against expectedFields instead of only validating shape ("improving" the harness without realizing it crosses the boundary). What happens: someone, seeing the fixtures file already has expectedFields, adds a field-by-field comparison between representativeModelResponse and expectedFields, thinking it makes the harness "more complete." How to spot it: if your version of run_smoke_test() ever reads the fixture's expectedFields key. How to fix it: reread lesson 5 — comparing against expectedFields is, precisely, the first step toward semantic evaluation, the territory this guide names but doesn't build. The field exists in the file so a human can read what extraction would be correct; this harness's code deliberately never touches it.

Interpreting Step 3's 3/5 as a signal about Nova Lite's quality, the model chosen in GENAI-COST-PROFILE.md (confusing the harness with a measurement of the real model). What happens: someone concludes "Nova Lite gets 60% of extractions right," based on this harness's result. How to spot it: if your interpretation of the 3/5 mentions Nova Lite, or any model, anywhere. How to fix it: none of this harness's data comes from Nova Lite or any model — all five representativeModelResponse are hand-written dictionaries, two of them deliberately broken so the harness has something to reject. The 3/5 measures, exclusively, that validate_shipment_fields() correctly classifies five deliberately built test candidates — it says absolutely nothing about how well Nova Lite, or any other model, would extract fields from a real manifest.

Forgetting the "REPRESENTATIVE" label when adding a new case to sample_manifests.json (the mistake test_every_fixture_has_a_representative_label exists to catch). What happens: someone adds a sixth case to the fixtures file, with its own representativeModelResponse, but forgets to write the word "REPRESENTATIVE" in the note field. How to spot it: Step 4's pytest fails, precisely, at test_every_fixture_has_a_representative_label — the error message points to exactly which case doesn't comply. How to fix it: this is precisely why that test exists as a code assertion, not just as a convention documented in prose — a human oversight gets caught automatically, the first time someone runs the complete suite, instead of quietly slipping into a data file no one reviews line by line again.

Exercises

Exercise 1 — Add, yourself, a sixth case to sample_manifests.json (shipment 4477) with a representativeModelResponse that has weightKg as a native JSON number (120) instead of a string ("120"). Before running the harness, predict whether that case would pass or fail, and why.

See solution

It would fail — validate_shipment_fields() (Module 4, lesson 6) marks weight_not_numeric as true if the value isn't a string, even if that value is a number: _is_numeric_string() starts with if not isinstance(value, str): return False, so a native JSON integer never passes that first check. The reported error would be "weightKg is not a numeric string". This exercise confirms, with your own case, exactly the same trap Module 4, lesson 6's Common Mistakes already warned about: ShipmentFields, as this guide defines it, requires all five fields to be strings, with no exception.

Exercise 2 — Explain why load_fixtures() doesn't need any special error handling for a malformed JSON file, unlike post_invoke_checks.py, which does explicitly validate with try/except json.JSONDecodeError. Is this an inconsistency, or a justified difference in context?

See solution

It's a justified difference, not an inconsistency. post_invoke_checks.py receives its JSON input from the command line (--json), a channel where a human can make a typo at any moment — validating and precisely reporting that error is, precisely, a well-designed command-line tool's job. load_fixtures(), on the other hand, reads a file committed to the repository, versioned, reviewed like any other code — malformed JSON there would be a programming error immediately caught by any test in this suite (test_five_fixtures_loaded() would fail with an exception before even reaching its assertion), not a real-time user input error. Adding exception handling there would be defensive code with no real use case to justify it — the same "don't add code for a problem that doesn't exist" discipline a careful engineer applies in any project.

Exercise 3 — Predict what would happen to Step 3's result (3/5) if someone fixed case 4473, adding "weightKg": "200" to its representativeModelResponse, without touching any other case. Would Step 5's guardrail block rate change too?

See solution

Step 3's result would go from 3/5 to 4/5 — 4473 would become PASS, because it would now have all five required keys with non-empty values and weightKg numeric. Step 5's block rate would change accordingly, from 2/5 (40%) to 1/5 (20%) — the two numbers are directly coupled, because both get calculated over the same set of run_smoke_test() results. This exercise confirms something important: the guardrail block rate isn't a fixed number for this guide — it depends, entirely, on which specific candidates enter the harness — exactly the same honesty this lesson's Step 5 already declared about the current 40%.

Summary and next step

This lesson built evals/manifest_extraction_smoke_test.py over evals/fixtures/sample_manifests.json — five cases, extending 4471/4472/4473 with two new ones —, and ran it for real: 3/5 fixtures pass structural validation, with the two failures — 4473 (missing field) and 4476 (unexpected field) — reusing, with no change, Module 4's validate_shipment_fields(). You verified the harness with six pytest cases, including one that guarantees no future case can sneak in without its "REPRESENTATIVE" label. You calculated, with real data over this specific batch, the guardrail block rate: 40%.

Before moving on you should be able to: explain why run_smoke_test() never reads expectedFields; recite the literal result (3/5, with the two exact failure reasons); and explain why this lesson's 3/5 says nothing about any real model's quality.

Lesson 7 documents, with the same exact honesty Module 3, lesson 6 and Module 4, lesson 7 already applied to apply and to the managed guardrail's blocking, this module's final boundary: why real inference latency, and the semantic quality this lesson's harness deliberately doesn't measure, can't be measured in this $0 lab — with the real CloudWatch metrics that would exist, if a real invocation happened.

Resources

  1. Google — SRE Workbook, Implementing SLOs — the source for the data-injection evaluation pattern lesson 5 already cited, applied here in code.
  2. This same course, Module 4, lesson 6 (06-hands-on-the-output-schema-validator.md) — the origin of validate_shipment_fields(), reused with no change in this lesson's Step 2.
  3. This same course, Module 1, lesson 3 — the origin of shipments 4471/4472/4473, extended in this lesson's Step 1 with 4475/4476.
  4. This same course, Module 7, lesson 5 (05-what-is-a-production-eval-and-why-it-is-not-a-unit-test.md) — the conceptual boundary this lesson builds in code.
← Previous5. What a production eval is, and why it isn't a unit testNext →7. The exact limit: why real latency and quality can't be measured here