Module 7: Observability Latency And Evals In Production
8. Project: Andes Cargo's AI workload observability dashboard
Description
This Module 7 final project turns the previous seven lessons into a single document: AI-OBSERVABILITY.md, at the root of andes-cargo-infra/, alongside SLO.md, which sre-and-incident-response-guide already left there — a complement, never a replacement, the exact same relationship GENAI-COST-PROFILE.md (Module 2, lesson 8) already established with COST-PROFILE.md. It brings together the three SLIs defined in lesson 2, lesson 4's real escalation-rate number, lesson 6's real guardrail block-rate number over the harness, and the exact latency and quality limit lesson 7 documented — with the same honesty ledger, section by section, every module project in this guide has already modeled.
Connection to the module
This document introduces no new data — every number it contains already ran, for real, in an earlier lesson of this module. Its job is exclusively integration and reconciliation: connecting lesson 4's 12.0% with the 10% GENAI-COST-PROFILE.md already declared as a hypothesis, and recording, in one place, exactly what in this observability layer is literal and what is representative, so Module 8 — this guide's capstone — can cite it without having to reread the previous seven lessons.
Step 1 — Why this document reconciles, doesn't replace, GENAI-COST-PROFILE.md
Before writing the document, it's worth resolving the same tension, with the same discipline, Module 2, lesson 8 already resolved for two different volume numbers. GENAI-COST-PROFILE.md, section 4, declared a 10% escalation-rate assumption (40 of ~400 manifests/month), explicitly labeled as a hypothesis pending revision "the first time a real escalation-rate number exists" — and that same section named, in advance, this module's lesson 4 as the source of that number. Lesson 4 already produced that number: 12.0%, over a fixed 50-event test batch.
THE RECONCILIATION THIS DOCUMENT RECORDS
GENAI-COST-PROFILE.md, section 4 AI-OBSERVABILITY.md, section 3
"10% -- hypothesis, pending "12.0% -- measured, over a fixed
revision" (Module 2, lesson 8) test batch" (Module 7,
lesson 4)
Both numbers are the SAME ORDER OF MAGNITUDE -- neither invalidates the
other. GENAI-COST-PROFILE.md, section 7, already declared the rule: this
triggers a REVISION, not a rewrite from scratch. That revision stays
PENDING -- outside this document's scope, which only records the
input data for when it happens.AI-OBSERVABILITY.md doesn't edit GENAI-COST-PROFILE.md — it leaves that document exactly as Module 2 left it, and records the 12.0% as the input data for the revision that, someday, someone with more real data would make.
Step 2 — Confirming both numbers, once more, before writing them
python3 observability/escalation_rate.py | grep "Escalation rate"
python3 evals/manifest_extraction_smoke_test.py | tail -1What to expect (literal — both scripts already ran in this module's lessons 4 and 6; this lesson only confirms the result is still identical before citing it in a new document):
Escalation rate 12.0%
Smoke test: 3/5 fixtures passed structural validation
Two deterministic scripts, each run a third time (the first when writing the original lesson, the second to verify them here), with the exact same result — the same determinism confirmation every script in this guide has already demonstrated in its own lesson.
Step 3 — The complete document
At the root of andes-cargo-infra/, create AI-OBSERVABILITY.md:
# AI-OBSERVABILITY.md — Andes Cargo AI Workload Observability Charter
**Status:** Active · **Owner:** Platform/SRE · **Framework:** Google SRE (SLI/SLO), extended to
`extract-shipment-manifest-fields`, the one Bedrock-backed escalation path defined in
`ADR-001-llm-as-escalation-path.md` (Module 1)
**Complements, never replaces:** `SLO.md` (`sre-and-incident-response-guide`, Module 2, lesson 8),
which covers the classic availability SLI of `process-shipment-manifest`. This document adds three
SLI specific to the AI escalation path, none of which existed before this guide.
**Does not cover:** semantic quality evaluation of any extraction (Module 7, lesson 5 -- AI
Engineering's job, not this guide's); a formal SLO target for any of the three SLI below (this
document declares SLI only -- fixing a defensible SLO needs more real data than a $0 lab produces).
## 1. Scope
This document declares the three SLI this guide adds for an AI workload, the real or representative
status of each, and the exact number this guide produced for the two that are calculable without
invoking Bedrock. It is the closing deliverable of Module 7, and the honesty ledger for the entire
observability surface this guide built.
## 2. The three SLI, formally
| SLI | Formula | Source lesson |
|---|---|---|
| Escalation rate | `ManifestParseFailed` events / total manifests processed | Module 7, lesson 2 |
| Guardrail block rate | candidates failing `validate_shipment_fields()` / candidates evaluated | Module 7, lesson 2 |
| Inference latency | invocations under a declared threshold / total invocations | Module 7, lesson 2 |
Escalation rate does not fit the classic "good events / valid events" mold (Module 7, lesson 2) --
it measures traffic composition, not success or failure. Guardrail block rate and inference latency
both fit the classic Google SRE formula for an SLI.
## 3. Escalation rate — measured, literal
`observability/escalation_rate.py` (Module 7, lesson 4), run against a fixed, deterministic batch of
50 test events, reusing `parse_manifest()` (Module 1, lesson 3) and `SHIPMENT_FIELDS_SCHEMA` (Module
4, lesson 6) unmodified:
```
Total test events 50
Escalated (ManifestParseFailed) 6
Escalation rate 12.0%
```
**This is the first real data point against `GENAI-COST-PROFILE.md` (Module 2, lesson 8, section 4)'s
10% assumption (40 of the ~400/month baseline `COST-PROFILE.md` declares).** Per that document's own
section 7 ("Document maintenance"), this number triggers a review, not a rewrite, of that assumption
-- both numbers are close in order of magnitude, and 50 fixed test events are explicitly not a month
of production traffic (the same caveat `sre-and-incident-response-guide`, Module 3, lesson 3 already
applied to its own 20-manifest verification batch). `GENAI-COST-PROFILE.md` is not edited by this
document -- it stays exactly as Module 2, lesson 8 left it, with this number recorded here as the
input for that future review.
## 4. Guardrail block rate — measured, on the smoke test harness
`evals/manifest_extraction_smoke_test.py` (Module 7, lesson 6), run against the five fixtures of
`evals/fixtures/sample_manifests.json`, reusing `validate_shipment_fields()` (Module 4, lesson 6)
unmodified:
```
[4471] PASS
[4472] PASS
[4473] FAIL - missing required field(s): weightKg
[4475] PASS
[4476] FAIL - unexpected field(s) not in ShipmentFields: confidenceScore
Smoke test: 3/5 fixtures passed structural validation
Guardrail block rate on this batch: 2/5 = 40%
```
**This number is real only on this specific five-fixture batch**, deliberately built to include two
distinct failure shapes (a missing field, an invented field) -- it is not a measurement of how often
a real Bedrock invocation would produce an incomplete response. What it does confirm, with executed
evidence: `validate_shipment_fields()` distinguishes complete from incomplete candidates with 100%
accuracy on any candidate it is given, real or representative -- the guarantee this SLI would need
the day real traffic exists to measure.
## 5. Inference latency — representative, no number
`InvocationLatency` and `TimeToFirstToken` (`AWS/Bedrock` namespace, milliseconds) are the real,
documented CloudWatch metrics that would populate this SLI -- confirmed against
`docs.aws.amazon.com/bedrock/latest/userguide/monitoring-runtime-metrics.html` (Module 7, lesson 7).
Neither has a value in this guide: no invocation of Nova Lite (the model `GENAI-COST-PROFILE.md`
chose) ever occurs. Module 7, lesson 7 confirmed, by direct search against official AWS sources, that
no published millisecond figure exists for Nova Lite specifically -- not even AWS's own
latency-optimized inference initiative covers this model. This SLI stays undeclared as a number,
by design, until a real invocation exists to measure it.
## 6. The eval boundary, restated
Module 7, lesson 5 drew the line this guide holds without exception: this guide builds the harness
that would run a production eval (Module 7, lesson 6); it never builds the semantic quality metric
that harness would need to actually judge correctness. `evals/fixtures/sample_manifests.json`
carries an `expectedFields` column specifically so a human reader can see what a correct extraction
would look like -- the harness itself never reads that column, precisely so it never crosses into
semantic evaluation. That boundary is AI Engineering's, not this guide's (Module 1, lesson 4).
## 7. Honesty ledger
| Component | Status | Evidence |
|---|---|---|
| `observability/structured_log.py`, the four-event vocabulary | Literal | Module 7, lesson 3 -- five log lines, run for real |
| `awslocal logs`/`awslocal cloudwatch` (metric filters, custom metrics) | Representative | No `LOCALSTACK_AUTH_TOKEN` in this sandbox (Module 7, lesson 3) |
| Escalation rate: 12.0% (6/50) | Literal | Module 7, lesson 4 -- deterministic, reproducible on any machine |
| Guardrail block rate: 40% (2/5) | Literal, on this specific batch | Module 7, lesson 6 -- never a production measurement |
| Inference latency | Representative, no number | Module 7, lesson 7 -- metric schema real, no invocation ever occurred |
| Semantic extraction quality | Out of scope by design | Module 7, lesson 5 -- AI Engineering's boundary |
## 8. Next steps
The moment a real AWS account with Bedrock access runs this guide's code end to end: (a) the
escalation rate script needs no changes -- point it at real `ManifestParseFailed` event history
instead of the fixed 50-event batch; (b) the smoke test harness needs no changes to its comparison
logic -- only `representativeModelResponse` in each fixture would be replaced with a real Bedrock
response, and the `note` field's `"REPRESENTATIVE"` label would need to come off, case by case, only
once that specific fixture's response is real; (c) `InvocationLatency`/`TimeToFirstToken` would begin
populating in CloudWatch automatically, with zero code changes, because Bedrock publishes both
natively for every real invocation. No part of this guide's code is throwaway lab code -- all of it
is the real instrumentation this workload would use in production, run here against fixed and
representative data because that is the exact, named limit of what this $0 lab can do.Step 4 — Verifying the document
wc -l AI-OBSERVABILITY.md
grep -c '^## ' AI-OBSERVABILITY.mdWhat to expect (literal — you wrote the content yourself, so its shape is deterministic):
119 AI-OBSERVABILITY.md
8
One hundred nineteen lines, eight sections — scope, the three SLIs, escalation, block rate, latency, the eval boundary, the honesty ledger, next steps. If your count isn't 8, check that you haven't merged or omitted any of Step 3's eight ## headings.
Why the document's "Step 8" matters as much as the rest
It's worth pausing on the document's last section, "Next steps" — it's easy, when writing an honesty document, to leave the impression that everything representative is a permanent limitation, with no way out. That section exists, specifically, to contradict that reading: every representative piece in this module is real production code, waiting for real data, not throwaway lab code. Lesson 6's harness doesn't need to be rewritten the day a real account exists — it just needs its representative dicts replaced with real responses, field by field, note by note. This distinction — "representative" doesn't mean "toy" — is the same one every previous module in this guide has already sustained for its own domain: Module 3's HCL wouldn't change if Bedrock were available on Hobby tomorrow; neither would Module 4's guardrail. This module closes that same promise for observability.
Common mistakes
Editing GENAI-COST-PROFILE.md directly from this lesson, instead of only recording the 12.0% in AI-OBSERVABILITY.md (jumping ahead to a revision this lesson doesn't authorize). What happens: someone, with the new number in hand, goes straight to GENAI-COST-PROFILE.md and changes its section 4's "10%" to "12%." How to spot it: if your diff for this project includes changes to a file from an earlier module. How to fix it: this lesson's Step 1 is explicit — this document records the input data for a future revision, it doesn't carry it out. GENAI-COST-PROFILE.md, section 7, describes what a real revision would change (the "Realistic" scenario would go from 40 to a new total, and bedrock_cost_estimate.py would need to run again) — real work, with its own cost implications, deserving its own conscious decision, not a one-word change made in passing in another module.
Presenting section 4's 40% block rate as if it were comparable, in the same sense, to section 3's 12.0% escalation rate (treating two "real numbers over a fixed batch" as if they carried the same evidentiary weight). What happens: someone reads both sections and treats them as equally representative of Andes Cargo's reality. How to spot it: if your summary of this document says "the escalation rate is 12% and the block rate is 40%, both confirmed." How to fix it: section 3's 50 events were designed to simulate, with variety, the kind of real traffic process-shipment-manifest would process — a broad sample, though small; section 4's 5 candidates were deliberately designed to include two specific types of failure, with the exclusive purpose of proving the validator catches them — not to simulate the real proportion of incomplete responses Bedrock would produce. The document already distinguishes this with the phrase "measured, on the smoke test harness" versus "measured, literal" — the distinction exists in the text, and it's worth keeping when citing either number outside this document.
Omitting section 8 ("Next steps") when summarizing this document for someone else, leaving the impression this module ended in a dead end (underestimating the value of what's already built). What happens: someone describes this project as "we documented everything we couldn't measure." How to spot it: if your summary of this document doesn't mention any of the three real code pieces this module leaves production-ready. How to fix it: this same lesson's "Why the document's 'Step 8' matters as much as the rest" section says it precisely — this module's work isn't a documentation-of-limits exercise, it's real observability infrastructure, with three pieces of executable code (structured_log.py, escalation_rate.py, manifest_extraction_smoke_test.py) that would work unchanged the day real traffic exists.
Exercises
Exercise 1 — Without looking at Step 3's document, list the three code pieces from this module the "Next steps" section declares as "real production code, not lab code." What would specifically change in each one the day a real Bedrock account exists?
See solution
observability/structured_log.py (Module 7, lesson 3) — nothing would change; the logger is already ready to instrument real invocations exactly as it instruments this guide's representative ones. observability/escalation_rate.py (Module 7, lesson 4) — only the data source would change, from a fixed 50-event batch to a real ManifestParseFailed event history, without touching would_escalate()'s logic. evals/manifest_extraction_smoke_test.py (Module 7, lesson 6) — the content of representativeModelResponse in each fixture would change, replaced with a real Bedrock response, without touching run_smoke_test()'s or validate_shipment_fields()'s logic. All three share the same pattern: the comparison/calculation logic is finished code; only the data it consumes would change.
Exercise 2 — Explain why the document's section 5 ("Inference latency") includes no table and no number, unlike sections 3 and 4, which do show a script's literal output. Why is the absence of a table, in itself, an honesty decision, not an accidental gap?
See solution
Including a table with an empty column, or with the word "N/A" repeated, would have suggested there's data pending completion — as if all that were missing was running one command to fill it in. Section 5's complete absence of a table communicates something more precise: there's no command this lab could run, not now nor with more time, to produce that number — the limitation is structural (no real invocation ever occurs in this guide, per Module 1's decision), not a pending task. It's the same discipline Module 7, lesson 7 already applied by writing "Average": "VARIABLE" instead of leaving the field empty or inventing a placeholder — naming the absence precisely, instead of leaving it ambiguous.
Exercise 3 — An external auditor reviews AI-OBSERVABILITY.md and asks: "why do you trust section 3's 12.0% if it's 'just' a 50-event test batch, not real traffic?" Write the answer the document, by its own logic, already supports.
See solution
An answer faithful to the document: "We don't trust the 12.0% as a measurement of Andes Cargo's real escalation volume — the document explicitly says so in section 3. We trust it as the first real, not hypothetical, evidence that the escalation criterion (would_escalate(), reusing parse_manifest() and SHIPMENT_FIELDS_SCHEMA unmodified) produces a number of the same order of magnitude as the 10% hypothesis GENAI-COST-PROFILE.md already declared months before this data existed. That doesn't prove Andes Cargo escalates exactly 12% of its real manifests — it proves the calculation mechanism works as designed, and gives a reasonable signal that the original hypothesis wasn't out of range. Real traffic measurement, if it ever happens, would replace this number — but it wouldn't invalidate the work of having calculated it with evidence, instead of with an assumption with no data behind it."
Summary and next step
This Module 7 final project wrote AI-OBSERVABILITY.md: eight sections declaring an AI workload's three SLIs (escalation, guardrail block rate, latency), with the two real numbers this module produced — 12.0% escalation over a fixed 50-event batch, 40% block rate over the five-fixture harness — and the exact latency and semantic-quality limit, with no invented number. You reconciled the 12.0% against the 10% hypothesis GENAI-COST-PROFILE.md already left declared, without editing that document, exactly as its own maintenance section described. You verified the document with wc and a section count, and confirmed, with the "Next steps" section, that no representative piece in this module is throwaway code.
Before moving on you should be able to: explain each of the document's eight sections' function; recite this module's two real numbers without looking at it; and explain why the document's "Next steps" is as important as its honesty ledger.
With AI-OBSERVABILITY.md closed, Module 7 in full — eight lessons, from sre-and-incident-response-guide's same SLI/SLO vocabulary to this dashboard — is behind us. Module 8, this guide's capstone, walks through the complete system end to end, including this module's escalation rate as the living proof of Module 1's thesis: a well-formed manifest never touches Bedrock.
Resources
sre-and-incident-response-guide, Module 2, lesson 8 (08-project-andes-cargos-slo-md.md) — the source forSLO.md, the sibling document this project complements, never replaces.- This same course, Module 2, lesson 8 —
GENAI-COST-PROFILE.md, section 7, the exact source for the promise this document's section 3 fulfills. - This module, lessons 2, 4, 6, and 7 — the complete source for every number and every limit in this document.
- FinOps Foundation — FinOps Framework, cited by
GENAI-COST-PROFILE.md— the same "declare hypotheses explicitly, revise with real data when it exists" spirit this document applies to observability.