Module 7: Observability Latency And Evals In Production
5. What a production eval is, and why it isn't a unit test
Description
This lesson answers a purely conceptual question, with not a single line of code, before lesson 6 builds the smoke test harness that needs it: what exactly is an eval — the term the industry uses for "evaluating an AI system's quality" — and how, precisely, does it differ from a unit test like the ones pytest already ran sixteen times in Module 4, lesson 6? The answer traces, again, the same boundary this guide's Module 1, lesson 4 already fixed with a verbatim quote from STRATEGY.md — this time applied specifically to quality evaluation, the last corner where that boundary could be crossed without anyone noticing.
Connection to the module
Lesson 4 closed this module's only 100% literal SLI. This lesson opens the ground where that honesty gets harder to sustain: measuring whether extract-shipment-manifest-fields extracts a manifest's fields well is, genuinely, the most interesting question about this AI workload — and also, precisely because of that, the one this guide has the most reason to name precisely and not try to answer. Lesson 6 builds the harness that would run that evaluation; this lesson carefully explains which part of that harness belongs to this guide and which doesn't.
Analogy: checking that a translation "sounds right" versus checking that code compiles
A unit test is like a spell checker: it verifies a text follows mechanical, objective, unambiguous rules — is the word spelled correctly? did the opening parenthesis get closed? does the function that got called exist? The answer is always binary, always the same for the same text, and any speaker of the language — or any compiler — would reach the same conclusion without argument. A quality eval, on the other hand, is like asking a bilingual human translator to read a translation and say whether it "sounds right" — not whether each individual word is spelled correctly (the spell checker already checked that), but whether the complete text conveys the original's nuance, tone, intent. Two expert translators, reading the same translation, could reasonably disagree about whether it "sounds right" — not because one is wrong, but because the question itself admits judgment, not just mechanical verification. post_invoke_checks.py (Module 4, lesson 6) is extract-shipment-manifest-fields's spell checker: it verifies the response has the correct shape, with a binary, indisputable answer. A semantic-quality eval is the bilingual translator's question: did this extraction really capture what the free-text manifest said? — and that question, this guide doesn't answer.
What an eval is, precisely
The industry uses "eval" (short for evaluation) as shorthand for a specific type of check: running an AI system against a set of test cases, and measuring how well the system's answers match what a human — or an explicit criterion designed to mimic human judgment — would consider correct. It's worth being precise about the mechanism, because sre-and-incident-response-guide, without using the word "eval," already described exactly this pattern, in a different context:
"[Correctness SLIs can be measured by] inject[ing] data with known outputs into the system, and count[ing] the proportion of times that the output matches our expectations."
Read it carefully: "inject data with known outputs, and count the proportion of times the output matches expectations" — is, literally, an eval's mechanics, described by Google SRE in the context of data-processing systems, years before "eval" became the trendy term for generative AI. The mechanics — a fixed dataset, comparison against an expected answer, a match percentage — are neither new nor exclusive to AI. What changes, and changes everything, is what "matches expectations" means when the output comes from a language model instead of a deterministic system.
Two kinds of "matching," and why they're different questions
SHAPE MATCHING MEANING MATCHING
(this guide, Modules 4 and 7) (AI Engineering)
"Does the response have "Is this specific response
ShipmentFields's five keys, with correct FOR THIS specific
non-empty values, numeric manifest? Did it capture the
weightKg?" right nuance of an ambiguous text?"
Verifiable with pure, Requires judgment -- human, or a
deterministic Python code, model acting as a "judge,"
no model involved which introduces ITS OWN
semantic uncertainty
The same candidate ALWAYS produces Two reasonable evaluators
the same PASS/FAIL result could disagree about the
SAME responseA concrete example makes the distinction impossible to confuse. Imagine two representative responses, both for the same free-text manifest from Module 1, lesson 3 (shipment 4471, "120kg of textiles, Lima to Santiago, AndesExpress"):
Response A: {"shipmentId": "4471", "originCountry": "Peru",
"destinationCountry": "Chile", "carrier": "AndesExpress",
"weightKg": "120"}
Response B: {"shipmentId": "4471", "originCountry": "Chile",
"destinationCountry": "Peru", "carrier": "AndesExpress",
"weightKg": "120"}Both pass post_invoke_checks.py with no problem — both have all five keys, every value is a non-empty string, weightKg is numeric. validate_shipment_fields() would mark both is_valid: True, exactly the same result. But Response B swapped origin and destination — a real extraction error, the kind only a semantic comparison against the original text (or against the expected result, expectedFields, if a dataset with that column existed) could catch. This is, as precisely as possible, the exact gap that separates "this guide" from "AI Engineering": Response B is a quality failure, invisible to any shape check, no matter how carefully written.
Why this guide doesn't cross that line, again
This guide's Module 1, lesson 4 already quoted, verbatim, STRATEGY.md's position within the NIEVA ecosystem graph: "Continuity from AI Engineering: the student arrives already knowing how to build AI systems. It doesn't repeat fundamentals: it teaches how to operate them." Evaluating whether a semantic extraction is correct — did the model understand that "Lima" is the origin and "Santiago" is the destination in this specific text, with this specific wording? — is, precisely, a skill of building AI systems with judgment, not of operating them. It's territory AI Engineering already covers in depth: how to design a set of evals with expected outputs verified by humans, how to use a model as a "judge" of another model, how to measure hallucination rates with statistical rigor. Repeating that content here, even if it were technically possible within this guide's $0 scope, would produce exactly the result Module 1, lesson 4 already warned against: a weaker version of something another ecosystem already teaches in depth.
What this guide does build — and it's real work, not an excuse to avoid the topic — is the harness: the test infrastructure that, the day someone connects a real Bedrock invocation, would automatically run against a fixed set of manifests and produce a report. An empty harness, with no infrastructure behind it, is useless the day a real account with Bedrock enabled actually exists. Building that harness now — with the comparison structure really working, even though the response it compares is representative — is genuine infrastructure work, exactly the same kind the rest of this guide already built for IAM, guardrails, and cost.
The table, revisited from Module 1, with a new row highlighted
| This guide DOES build | This guide does NOT build — it's AI Engineering |
|---|---|
| … (seven rows already established in Module 1, lesson 4) | … |
| The smoke test harness — comparison structure, not the quality metric — M7.6 | The semantic-quality metric that harness would compare in real production |
This is, literally, the last row of the table Module 1, lesson 4 already built in full — Module 7 doesn't add a new table, it closes the one that already existed with the one piece that still needed to be named in detail.
Common mistakes
Thinking "building the harness" is an indirect way of doing quality evaluation after all, just without saying so (underestimating the boundary's precision). What happens: someone argues that, since lesson 6's harness is going to compare a response against expectedFields, it's actually doing semantic evaluation, just under a different name. How to spot it: if your description of lesson 6's harness uses the phrase "verifies the extraction is correct" without the word "shape" or "schema" somewhere nearby. How to fix it: revisit this lesson's Response A and B example — lesson 6's harness, as designed, would approve both, because both have the correct shape. A harness that did semantic evaluation would reject Response B for swapping origin and destination; the one this guide builds, deliberately, doesn't. That is, exactly, the line.
Concluding an eval is "nothing more than" a unit test with different data, with no real underlying difference (underestimating why the industry uses a new term). What happens: someone argues "eval" is just a trendy word for what pytest has already been doing for years. How to spot it: if your explanation of "eval" doesn't mention, anywhere, that the response under test comes from a non-deterministic system. How to fix it: the comparison mechanics — a fixed dataset, counting matches — really are the same, this lesson's Google SRE quote confirms it. The real difference is that a unit test checks deterministic code, where "correct" has a single, objective answer verifiable with logic; an eval checks a model's output, where "correct" often admits more than one reasonable answer, and the comparison frequently needs human judgment or a model acting as judge — a structurally different type of check, not just a marketing vocabulary difference.
Looking in this lesson for a definition of "how good is good enough" for an Andes Cargo extraction (expecting a quality threshold this guide can't give). What happens: someone, finishing this lesson, asks "so what accuracy percentage would extract-shipment-manifest-fields need to be acceptable in production?" How to spot it: if your question after this lesson looks for a minimum acceptable quality number. How to fix it: that question — fixing an acceptable semantic-quality threshold, and measuring against it — is, with exact precision, AI Engineering territory; this guide has no basis to answer it, because it never invokes the real model that would produce the data needed to calculate that rate. What it can honestly say: the infrastructure to run that measurement, the day it exists, is built in lesson 6.
Exercises
Exercise 1 — Using this lesson's Response A and B example, design a THIRD representative response that fails post_invoke_checks.py (Module 4, lesson 6) but that, had it been correct in shape, would also have had a different semantic error from Response B's. What does this demonstrate about the independence of the two types of check?
See solution
For example: {"shipmentId": "4471", "originCountry": "Peru", "destinationCountry": "Chile", "carrier": "AndesExpress"} — no weightKg, so post_invoke_checks.py rejects it (missing required field(s): weightKg). But imagine that, on top of that, this same response had written "carrier": "RutaSur" instead of "AndesExpress" (a semantic error — the original manifest does mention AndesExpress). This exercise's point: the two types of failure — shape and meaning — are completely independent. A response can fail only in shape (like this one), only in meaning (like Response B), both at once, or neither. post_invoke_checks.py can only detect the first type, no matter how serious the second one is — the clearest possible demonstration of why a shape check never substitutes for a quality evaluation, and vice versa.
Exercise 2 — Explain, in your own words and without repeating this lesson's analogy, why two reasonable human evaluators could disagree about whether Response B (origin/destination swapped) is "a serious failure" or "a minor error." What kind of business decision would determine which of the two positions is correct for Andes Cargo?
See solution
They could disagree because how serious it is to swap origin and destination depends on what the system does after that extraction — a business question, not a linguistic one. If Shipments feeds, say, a customs billing system that charges differently based on the declared country of origin, swapping Peru and Chile could generate an incorrect declaration with real legal consequences — a serious failure. If, instead, the field is used only for an internal volume report a human reviews before acting, the same error might be fixable with a simple manual review — a minor error, though still an error. This is precisely the kind of decision a real eval would need as an explicit criterion before it could rate any response as "acceptable" or not — and it's, with the same precision, exactly the kind of product decision this guide deliberately doesn't make, because it belongs to whoever designs the AI system, not whoever operates it.
Exercise 3 — A coworker, after reading this lesson, proposes: "let's give this harness to the model itself, and have the model evaluate whether its own answers are correct — that way we close the loop without needing AI Engineering." What problem does that proposal have, even if it were technically possible within this guide?
See solution
The underlying problem isn't technical feasibility — using a model as a "judge" of another model's (or the same model's) responses is, in fact, a real, documented technique, mentioned in passing in this lesson's definition of "eval." The problem is that "using an LLM as a judge, with a prompt designed to evaluate semantic quality with judgment" is, precisely, the kind of advanced prompt engineering technique this guide's Module 1, lesson 4 already explicitly named as AI Engineering territory, not this guide's. Building that piece here wouldn't "close the loop without needing AI Engineering" — it would, with exact precision, mean starting to build AI Engineering inside a guide that deliberately exists not to. The correct answer to your coworker's proposal: "that's a real idea, and the right place to learn it properly is the AI Engineering ecosystem, not here."
Summary and next step
This lesson defined, precisely and with no code, what an eval is — "inject data with known outputs, count the proportion of matches," cited from the same Google SRE source that has already sustained the rest of this guide — and why that mechanic, applied to a language model, splits into two completely independent questions: does the response have the correct shape? (this guide) and does the response have the correct meaning? (AI Engineering). You saw, with a concrete example — origin and destination swapped —, a response that passes any shape check without a problem, while seriously failing on meaning, the clearest possible demonstration of why one check never substitutes for the other.
Before moving on you should be able to: cite, from memory or close to it, Google SRE's definition of "eval" and explain why it applies here; build, unaided, an example of a response that passes shape but fails meaning; and explain why this guide never builds the "meaning" half, even when it would be technically possible within its $0 scope.
Lesson 6 builds the real harness: evals/manifest_extraction_smoke_test.py, over evals/fixtures/sample_manifests.json — the comparison structure really runs, against precisely labeled representative candidates, exactly on the side of the line this lesson just drew.
Resources
- Google — SRE Workbook, Implementing SLOs — the exact source for the definition of evaluation by injecting data with known outputs, cited in this lesson.
- This same course, Module 1, lesson 4 (
04-the-boundary-with-ai-engineering-said-out-loud.md) — the complete boundary-with-AI-Engineering table, which this lesson closes with its last row. - This same course, Module 4, lesson 6 (
06-hands-on-the-output-schema-validator.md) — the origin ofpost_invoke_checks.py, the "shape matching" piece this lesson contrasts with semantic evaluation. src/paths/aws-cloud-ecosystem/STRATEGY.md— the complete source for this guide's position in the NIEVA ecosystem graph, first cited in Module 1, lesson 4, revisited here.