Module 7: Observability Latency And Evals In Production
1. Module introduction: the same vocabulary, new metrics
Description
sre-and-incident-response-guide already left, finished and verified, the complete vocabulary this module is going to use without re-explaining it from scratch: what an SLI is, what an SLO is, how an error budget gets calculated, how to read AWS/Lambda/Errors with awslocal cloudwatch get-metric-statistics, and the "valid events, good events" pattern that turns an intuition ("the system seems fine") into a defensible number. This module doesn't repeat any of that. It reapplies it, with the exact same discipline, to a different workload — one that invokes a language model, not just a deterministic Lambda — and asks the question no previous guide in this ecosystem had to answer yet: how much of all this can actually be measured, without ever invoking Bedrock once?
Connection to the module
This guide's Modules 1 through 6 built, piece by piece, extract-shipment-manifest-fields's infrastructure, guardrails, and cost model. This module closes the operational circle with the third discipline every production workload needs, alongside security (M5) and cost (M6): observability. Lesson 2 defines the three new indicators; lessons 3 through 6 build them, each with its own exact honesty about what really ran; lesson 7 draws the final boundary, the same kind of boundary M3.6 and M4.7 already drew for apply and for the guardrail's blocking; lesson 8 closes the module with the documented dashboard.
Analogy: the same instruments, a different engine
A mechanic who knows how to read a combustion car's dashboard — speedometer, tachometer, engine temperature, fuel level — doesn't have to learn to read new instruments the day they sit in front of an electric car. There's still a speedometer. There's still an indicator for "how much energy is left." What changes isn't the type of instrument — it's what specific figure appears behind the needle, and what a high or low value means for this specific engine. sre-and-incident-response-guide taught you to read the complete dashboard of a production system: SLI (the needle), SLO (the speedometer's green zone), error budget (how much margin you have left before leaving that zone). This module changes no instrument. It changes the engine — from a Lambda that parses key=value to a Lambda that, when that parsing fails, invokes a language model — and teaches you to read the new figures that specific engine produces, with the same tools as always.
THE DASHBOARD (sre-and-incident-response-guide) THE NEW ENGINE (this module)
SLI = a quantitative measure of some Escalation rate
aspect of the level of service Inference latency
SLO = the numeric target on that SLI Guardrail block rate
Error budget = 100% - SLO, in minutes
The same three terms from
Already built, already verified -- the left, applied to numbers
process-shipment-manifest, Modules 1-4 no previous guide in this
of that guide ecosystem had to measureWhat this module inherits, without re-explaining it
| Piece | Origin | What this module does with it |
|---|---|---|
| The SLI formula: "the proportion of valid events that were good" | sre-and-incident-response-guide, Module 2, lesson 3, citing Google — The Art of SLOs | Reapplied in lesson 2, with the honesty that one of the three new SLIs does not cleanly fit that mold |
| SLI, SLO, error budget — the exact definitions | sre-and-incident-response-guide, Module 1, lesson 7 (07-the-vocabulary-youll-use-all-guide.md), citing the Google SRE Book | Taken as known — this lesson doesn't cite them again at length, lesson 2 does whenever each new SLI needs it |
| The real CloudWatch Logs/Metrics pattern over an inherited Lambda | sre-and-incident-response-guide, Module 3, lessons 3 and 4 | This module's lesson 3 applies the same pattern to extract-shipment-manifest-fields, with new structured logging |
The finding that awslocal cloudwatch/awslocal logs are representative in this specific environment (no LOCALSTACK_AUTH_TOKEN) | sre-and-incident-response-guide, Module 3, lessons 3 and 4 | Reconfirmed, without re-investigating it, in this module's lesson 3 |
parse_manifest(), the deterministic parser | This same course, Module 1, lesson 3 | Reused, with no change, as lesson 4's calculation engine |
SHIPMENT_FIELDS_SCHEMA, validate_shipment_fields() (guardrails/post_invoke_checks.py) | This same course, Module 4, lesson 6 | Reused, with no change, in this module's lessons 4 and 6 |
| The boundary with AI Engineering | This same course, Module 1, lesson 4 | Explicitly revisited in lesson 5, this time applied to quality evaluation |
Notice the pattern: four of this table's seven rows aren't from sre-and-incident-response-guide — they're from this same guide, from earlier modules. This module doesn't just inherit vocabulary from a sibling guide; it inherits real code, already built and already tested, from its own Modules 1 and 4. No SLI in this module starts from scratch.
Map of the eight lessons
M7.1 Introduction (this lesson) -- the same vocabulary, new metrics
M7.2 The three SLIs of an AI workload -- formal definitions, with
the honesty of which fits the classic mold and which doesn't
M7.3 Hands-on: the extractor's CloudWatch Logs/Metrics --
EXECUTED (structured logging), REPRESENTATIVE (awslocal)
M7.4 Hands-on: the escalation rate metric --
EXECUTED, this guide's only 100% literal AI SLI
M7.5 What a production eval is, and why it isn't a unit test
-- conceptual, the boundary with AI Engineering, again
M7.6 Hands-on: the smoke test harness with fixed manifests
-- EXECUTED (the harness), REPRESENTATIVE (the model's output)
M7.7 The exact limit: why real latency and quality
can't be measured here -- REPRESENTATIVE, with the exact reason
M7.8 Project: Andes Cargo's AI workload observability
dashboard -- EXECUTED (the document)Lessons 3, 4, and 6 are hands-on; each one makes clear, right at the moment each code block appears, whether what you're seeing really ran or is a precise reconstruction of what would run. Lesson 5 is the module's only purely conceptual one, and it exists because the line it draws — formal correctness versus semantic quality — is what sustains lesson 6's honesty. Lesson 7 closes with the same kind of honesty M3.6 and M4.7 already modeled: not a new obstacle, the same underlying limit, this time applied to time and quality.
This module's honesty ledger, in advance
Before going into detail, it's worth seeing the complete table all at once — the same explicit-honesty discipline every module in this guide has already applied to its own domain:
| SLI / piece | Can it be calculated without invoking Bedrock? | State in this module |
|---|---|---|
Escalation rate (ManifestParseFailed / total) | Yes — depends only on parse_manifest() and which manifests arrive, never on a model invocation | Literal (M7.4) |
| Custom guardrail block rate | Yes — depends on validate_shipment_fields(), custom code, run over any candidate, real or representative | Calculable, over M7.6's harness data |
| Inference latency | No — InvocationLatency/TimeToFirstToken (CloudWatch, AWS/Bedrock) only exist if a real invocation happened | Representative (M7.7) |
| Extraction's semantic quality | No, and also out of this guide's scope by design (M1.4) | Named, not built (M7.5) |
| The handler's structured logging | Yes — it's custom Python code, with no dependency on Bedrock whatsoever | Literal (M7.3) |
| CloudWatch metrics over that logging | Only with LOCALSTACK_AUTH_TOKEN, unavailable in this environment | Representative (M7.3) |
Three literal rows, two representative, one out of scope by design. That is, precisely, the complete map of what this module can and can't demonstrate — and each of the eight lessons that follow states, at the exact moment it's relevant, which of those rows it falls into.
Common mistakes
Expecting this module to teach configuring Prometheus/Grafana for Bedrock, the way sre-and-incident-response-guide already did for process-shipment-manifest (a tooling expectation). What happens: someone, familiar with the sibling guide's Module 3, looks here for a docker-compose.yml with a Bedrock metrics exporter. How to spot it: if your first question upon opening this module is "where's the Grafana dashboard?" How to fix it: this module deliberately stays at native CloudWatch — the same scope sre-and-incident-response-guide, Module 3, lesson 3 already used for Lambda's native metrics, before reaching Prometheus in its lesson 6. Standing up a complete observability stack for a workload that, honestly, never invokes a real model in this lab would be building infrastructure for data that doesn't exist yet.
Assuming "three new SLIs" means all three get calculated with the same kind of formula (over-generalizing from sre-and-incident-response-guide). What happens: someone, used to the sibling guide's "good events ÷ valid events" formula, tries to mechanically apply it to this module's three SLIs without thinking about whether it fits. How to spot it: if your definition of escalation rate uses the word "good" anywhere. How to fix it: this module's lesson 2 dedicates a complete section to this exact distinction — escalation rate doesn't measure "what proportion of invocations was good," it measures "what proportion of total traffic took the expensive path." These are different questions, and confusing them is precisely the mistake lesson 2 carefully prevents.
Concluding, from this lesson's honesty table, that "half this module is useless for anything real" (underestimating what is literal). What happens: someone sees two rows marked "representative" and dismisses the entire module as mostly theoretical. How to spot it: if your summary of this module is "you can't measure almost anything without paying for Bedrock." How to fix it: the escalation rate — this guide's most important SLI for deciding whether the AI path stays a minority of traffic, M1.6's central thesis — is 100% literal, run over a real set of events. The structured logging that instruments the handler is real, executed, tested Python code. What stays representative is, specifically, what depends on a real model invocation — the exact same limit every previous module in this guide already declared for its own domain, never a new limitation specific to this module.
Exercises
Exercise 1 — Without looking at this lesson's honesty table, predict which of the three new SLIs (escalation rate, inference latency, guardrail block rate) is this module's only 100% literal one. Justify your answer using what you already know from this guide's Module 1.
See solution
The escalation rate. The reason, already established since Module 1, lesson 3: ManifestParseFailed gets published — or not published — exclusively based on whether parse_manifest() produced ShipmentFields's five fields, a decision made before any attempt to invoke Bedrock exists. Inference latency, by definition, doesn't exist without a real invocation (there's nothing to time). The guardrail block rate is halfway there: it's calculable with custom code (validate_shipment_fields()), but needs a candidate — real or representative — to run over; it isn't as self-sufficient as the escalation rate, which only needs the manifest's raw text.
Exercise 2 — Explain, in your own words, why this module can reuse sre-and-incident-response-guide's SLI formula without having to redefine SLI, SLO, or error budget from scratch. What would make that reuse a mistake, instead of a correct decision?
See solution
Reusing the formula is correct because SLI, SLO, and error budget are concepts independent of the type of system they measure — Google SRE's definition ("a carefully defined quantitative measure of some aspect of the level of service") mentions no Lambdas, LLMs, or any specific type of workload. It would be a mistake, on the other hand, to assume that process-shipment-manifest's specific definition of an SLI — for example, "with no exception, within the timeout, with a correct record in Shipments" — carries over unchanged to extract-shipment-manifest-fields. The vocabulary gets inherited; each SLI's specific content doesn't — exactly the distinction this module's lesson 2 carefully develops for each of the three new SLIs.
Exercise 3 — Based on this lesson's honesty ledger, predict what would happen to the "inference latency" row if, someday, someone ran this complete guide against a real AWS account with Bedrock enabled. Would it automatically change from representative to literal, or would something more be needed?
See solution
It would change to literal, but not automatically — it would take, at minimum, actually executing the invocations this guide never executes, and reading InvocationLatency/TimeToFirstToken from CloudWatch (AWS/Bedrock) after those invocations exist. This guide's code and commands — the ones lesson 7 precisely documents against those metrics' real schema — would stay exactly the same; the only thing that would change is that, for the first time, real traffic would exist behind them. It's the same relationship M3.6 already established for apply against Bedrock: the code doesn't change between the $0 lab and a real account, what changes is whether the service behind that code exists.
Summary and next step
This lesson mapped this module's eight lessons and confirmed, table by table, exactly what it inherits from sre-and-incident-response-guide (the SLI/SLO/error budget vocabulary, the CloudWatch pattern, the finding that awslocal is representative in this environment) and what it inherits from this guide's own Modules 1 and 4 (parse_manifest(), SHIPMENT_FIELDS_SCHEMA, validate_shipment_fields()). You saw, in advance, this module's complete honesty ledger: three literal pieces, two representative, one out of scope by design.
Before moving on you should be able to: name this module's three new SLIs without help; explain why the SLI formula gets reused unchanged but each SLI's specific content doesn't; and predict which of the three is the only 100% literal one, with its justification.
Lesson 2 formally, precisely defines each of the three new SLIs — and takes special care to explain why the escalation rate doesn't cleanly fit the classic "good events ÷ valid events" mold sre-and-incident-response-guide already established.
Resources
- Google SRE Book, Chapter 4 — Service Level Objectives — the exact source for SLI/SLO/SLA, already cited by
sre-and-incident-response-guide, Module 1, lesson 7. - Google — The Art of SLOs (Participant Handbook) — the exact SLI formula ("the proportion of valid events that were good"), carefully reapplied in this module's lesson 2.
sre-and-incident-response-guide, Module 3, lessons 3 and 4 — the exact precedent for real CloudWatch Logs/Metrics over an inherited Lambda, and the finding thatawslocal cloudwatch/awslocal logsare representative withoutLOCALSTACK_AUTH_TOKEN.- This same course, Module 1, lesson 3 (
03-andes-cargos-ai-workload-when-the-deterministic-parser-is-not-enough.md) — the origin ofparse_manifest()and of the escalation rate as a concept. - This same course, Module 4, lesson 6 (
06-hands-on-the-output-schema-validator.md) — the origin ofSHIPMENT_FIELDS_SCHEMAandvalidate_shipment_fields(), reused in this module's lessons 4 and 6.