Module 7: Observability Latency And Evals In Production

2. SLIs for an AI workload: escalation, latency, guardrail block rate

Description

This lesson defines, with the same formal precision sre-and-incident-response-guide, Module 2, lesson 3 already applied to process-shipment-manifest, extract-shipment-manifest-fields's three new SLIs: the escalation rate, inference latency, and the custom guardrail block rate. None of the three gets invented here — each has an exact formula, a cited source, and an explicit decision on what counts as the numerator and what counts as the denominator. And one of the three, quite deliberately, doesn't cleanly fit the classic SLI mold the sibling guide already established — this lesson explains why, instead of forcing the formula where it doesn't belong.

Connection to the module

Lesson 1 promised three new SLIs and previewed the complete honesty ledger. This lesson fulfills that promise with the three formal definitions. Lessons 3 through 6 build, one by one, the code each definition needs to produce a real number — the escalation rate in lesson 4, the block rate in lesson 6 (as a byproduct of the smoke test harness), latency never in executed code, only precisely documented in lesson 7.

Analogy: three needles on the same dashboard, three different kinds of question

Revisit lesson 1's electric car dashboard. Not every needle on a dashboard measures the same kind of thing. The speedometer answers "how fast am I going, right now?" — an instantaneous value. The "kilometers until the battery runs out" indicator answers a different question: "how much capacity do I have left, given my current consumption rate?" — a projection. And a binary warning light ("check engine: yes/no") answers a third, still different question: "did something happen that requires attention?" All three are real information about the same vehicle, but none is interchangeable with the other two. This lesson's three SLIs work the same way: the escalation rate answers "what proportion of my total traffic is taking the expensive path?" — a traffic composition question, not one of success or failure. Inference latency answers "how fast does the model respond when I do invoke it?" — a performance value. The guardrail block rate answers "what proportion of responses that reach my schema validator pass the check?" — the only one of the three that does cleanly fit the classic "good events ÷ valid events" mold.

SLI 1 — Escalation rate: a composition question, not a success question

The formula

   Escalation rate = ManifestParseFailed events / total manifests processed

Numerator and denominator, precisely:

Term	Exact definition
Total manifests processed	Every `process-shipment-manifest` invocation triggered by a real manifest upload to `andes-cargo-shipment-docs` — the exact same exclusion `sre-and-incident-response-guide`, Module 2, lesson 3 already established for manual test invocations: they never count
`ManifestParseFailed` events	The subset of those manifests where `parse_manifest()` (Module 1, lesson 3) failed to produce `SHIPMENT_FIELDS_SCHEMA`'s five fields (Module 4, lesson 6) — the exact criterion that triggers the event's publication to `andes-cargo-events`

Why this formula is NOT "good events ÷ valid events"

Here's the distinction worth carefully developing, because it's easy to force it without noticing. The SLI formula sre-and-incident-response-guide cited from Google is "the proportion of valid events that were good" — the proportion of valid events that were good. That formula assumes a clear notion of "good" versus "bad" exists: an invocation that ends well is good, one that fails is bad. Applied to process-shipment-manifest, that works perfectly — the sibling guide already demonstrated it.

But a ManifestParseFailed isn't a bad event. It's, literally, the system working exactly as designed: process-shipment-manifest correctly detected that this manifest doesn't have the expected key=value format, and published an honest event so the escalation path can try it. If we treated every ManifestParseFailed as a "bad event" in a classic SLI formula, we'd be measuring, without noticing, something that sounds like "reliability" but is actually "how predictable is our logistics partners' input format" — a business question, not a systems engineering one. The escalation rate is, instead, a traffic composition SLI: it tells you how much exposure Andes Cargo has to the expensive, non-deterministic path, without judging whether that's "good" or "bad" in itself.

   CLASSIC SLI (availability)                THIS MODULE'S SLI (composition)

   What proportion of invocations            What proportion of total traffic
   ended WELL?                               took the EXPENSIVE path?

   "Good" and "bad" have a clear             There's no "good" or "bad" -- there's
   meaning ahead of time                     "cheap" and "expensive," and both are
                                              valid outcomes of the system
   A typical SLO: 99.9% good
                                              A typical SLO: a CEILING --
                                              "no more than X% should escalate,"
                                              an alert signal if crossed,
                                              not a perfection target

Why this still matters for reliability, without being a reliability SLI

It's worth not taking this distinction too far: even though the escalation rate doesn't measure "good ÷ bad," it is a real operational signal, with the same practical implications as any SLI. A sudden jump — from a usual 12% to 60% overnight — is exactly the kind of anomaly that would warrant an alert, the same discipline sre-and-incident-response-guide, Module 4 already built for burn rate. The difference isn't "this matters less" — it's that the reason for a high value isn't "the system is broken," it's "something in the real world changed" (a large logistics partner switched formats, for example) — the exact same distinction this guide's Module 1, lesson 3, Exercise 3 already previewed.

SLI 2 — Inference latency: the classic mold, with no data to fill it

The formula, cited from the source

"The proportion of requests that were faster than some threshold."

— Google — SRE Workbook, Implementing SLOs

Translated to extract-shipment-manifest-fields:

   Latency SLI = invocations with InvocationLatency <= declared threshold
                 -------------------------------------------------------
                 total invocations that reached Bedrock

This formula does cleanly fit the classic mold — it's a "proportion of good events" SLI in the strict sense: a "good event" is an invocation that responds within the declared time budget. The problem isn't the formula. It's that InvocationLatency — the real metric Amazon Bedrock publishes to CloudWatch, under the AWS/Bedrock namespace, with a ModelId dimension — only has a value when a real invocation happened. With not a single invocation, there's no data at all to calculate either this formula's numerator or its denominator. This module's lesson 7 develops this with complete precision, including the real metrics Bedrock publishes and why none of them has a value in this $0 lab.

SLI 3 — Custom guardrail block rate: the other SLI that does fit the classic mold

The formula

   Block rate = candidates that fail validate_shipment_fields()
                --------------------------------------------------
                total candidates evaluated by post_invoke_checks.py

validate_shipment_fields() (guardrails/post_invoke_checks.py, Module 4, lesson 6) already produces, for any candidate — real or representative —, a binary result: is_valid True or False. This means this SLI does fit Google's classic formula: a "good event" is a candidate that passes validation (PASS, writable to Shipments); a "valid event" is any candidate that reached the validator, regardless of where it came from. Unlike the escalation rate (SLI 1), here "good" and "bad" do have an unambiguous meaning: a candidate that fails schema validation, by definition, can't be written to Shipments without corrupting the contract write_shipment_record() expects.

   Block rate = 1 - (PASS candidates / evaluated candidates)

The difference from SLI 2 (latency) is just as important: this SLI is calculable without invoking Bedrock, because validate_shipment_fields() doesn't care where the candidate it's evaluating came from — it only cares, exclusively, about whether it has the correct shape. Any set of precisely hand-built representative candidates (like the ones M7.6 is going to build) produces a real block-rate number, with the same labeling honesty every representative datum in this guide already carries: the number is real for those specific candidates, never a measurement of how often Bedrock, in production, would produce an incomplete response.

The three SLIs, in a quick-reference table

SLI	Formula	Fits "good/valid"?	Calculable without Bedrock?	Where it's calculated in this guide
Escalation rate	`ManifestParseFailed` / total processed	No — it's composition, not success/failure	Yes, always	M7.4 (literal)
Inference latency	invocations under threshold / total invoked	Yes, in theory	No — depends on a real invocation	M7.7 (named, no data)
Guardrail block rate	`FAIL` candidates / evaluated candidates	Yes	Yes, over any candidate, real or representative	M7.6 (over the harness)

Common mistakes

Treating a ManifestParseFailed as an "error" in the same sense as an uncaught KeyError (confusing composition with failure). What happens: someone, seeing the escalation rate go up, reacts as if the system were failing, the same way they'd react to a jump in AWS/Lambda/Errors. How to spot it: if your first reaction to an increase in the escalation rate is "we need to fix the bug." How to fix it: a ManifestParseFailed is, by design, the system's correct behavior facing a manifest that doesn't have the expected format — this lesson's section on "why this formula isn't 'good events ÷ valid events'" develops exactly this distinction. A sustained increase deserves investigation (did something in the real world change?), not a code fix as the first hypothesis.

Trying to calculate the guardrail block rate over real Bedrock invocations, instead of over any available candidate (underestimating what validate_shipment_fields() needs). What happens: someone concludes this SLI, like latency, also needs a real invocation to be calculated. How to spot it: if your plan for measuring the block rate includes the phrase "first I need Bedrock to actually respond." How to fix it: validate_shipment_fields() is pure Python code, with no dependency on Bedrock whatsoever — it accepts any dict, wherever it came from. This module's M7.6 is going to build a set of representative candidates, precisely to demonstrate this SLI does produce a real number without needing any invocation.

Looking for an explicit SLO for this lesson's three SLIs, and getting frustrated not finding it (a scope expectation). What happens: someone, used to sre-and-incident-response-guide, Module 2 defining both the SLI and the SLO for every metric, looks in this lesson for a target number ("the escalation rate should be under X%"). How to spot it: if your question after this lesson is "so what's the SLO?" How to fix it: this module deliberately stops at defining the SLI — fixing a meaningful SLO for a workload with so little real data (this guide never invokes Bedrock) would mean inventing a number without the evidence sre-and-incident-response-guide, Module 2, lesson 5 required before choosing any real SLO. M7.4 does produce a real escalation-rate number — the first honest step toward, eventually, being able to fix a defensible SLO, not the SLO itself.

Exercises

Exercise 1 — Classify each of this lesson's three SLIs by whether "higher" is better, worse, or neither on its own. Justify each classification with this lesson's exact definition.

See solution

Escalation rate: neither on its own — it isn't a "higher is better" nor "lower is better" SLI in a moral sense; it's a traffic-composition signal. A stable value is what's expected; a sudden jump is the signal, not the absolute value. Inference latency: lower (faster, more invocations under the threshold) is better, exactly like any classic latency SLI. Guardrail block rate: lower is better — a block means an extraction couldn't be written to Shipments, so fewer blocks, over the same volume, is a positive signal. Notice that two of the three do fit the simple "lower is better" intuition; only the escalation rate breaks that intuition, exactly this lesson's central point.

Exercise 2 — A colleague proposes: "let's measure the escalation rate with the same availability formula as process-shipment-manifest: let's treat ManifestParseFailed as a 'bad event' and calculate 1 minus that proportion as our 'parser health' SLI." What problem does that proposal have, beyond the label?

See solution

The problem isn't just the name — it's that formula would measure, with perfect mathematical precision, something that isn't process-shipment-manifest's to measure: the variability of Andes Cargo's logistics partners' input format, an external factor, outside the code's control. A "health" SLI implicitly suggests a low value requires a system fix — but the real fix, if the escalation rate goes up, is almost never "fix parse_manifest()" (that function fulfills its contract perfectly, per Module 1, lesson 3), it's investigating what changed on the logistics partner's side, or deciding, with data, whether it's worth expanding what the parser recognizes directly. Naming this "health" would bias the investigation toward the wrong answer from the very first moment.

Exercise 3 — Explain why the guardrail block rate (SLI 3) can be calculated today, with real data, while inference latency (SLI 2) can't, even though both use the same classic "good events ÷ valid events" formula. What distinguishes the data each one needs?

See solution

The difference is in what produces each formula's numerator and denominator. The block rate needs candidates — Python dictionaries with ShipmentFields's keys — and validate_shipment_fields() is indifferent to where those dictionaries came from: it can evaluate a hand-written one exactly the same way it would evaluate one that came from a real invocation, because its job is checking shape, not origin. Latency, on the other hand, measures the time a specific invocation took — data that, by definition, only exists if that invocation really happened; there's no way to "hand-build" a latency value without inventing it, and this guide, with the same discipline as its entire honesty section, refuses to present an invented number as if it were a real measurement.

Summary and next step

This lesson defined, with formula and cited source, this module's three new SLIs: the escalation rate (traffic composition, not good/bad, calculable without Bedrock), inference latency (the classic Google SRE mold, with no real data to fill it in this lab), and the custom guardrail block rate (also the classic mold, and calculable, because validate_shipment_fields() doesn't depend on any real invocation). You carefully saw why the escalation rate is the only one of the three that breaks the simple "good events ÷ valid events" intuition — and why that doesn't make it less important as an operational signal.

Before moving on you should be able to: write the three SLIs' exact formulas without help; explain why a ManifestParseFailed isn't a "bad event"; and predict which of the three are calculable today, with no real Bedrock invocation.

Lesson 3 returns to executable ground: extract-shipment-manifest-fields's real structured logging, and the CloudWatch metrics — real in their definition, representative in this specific environment — that would feed, in production, each of this lesson's three SLIs.

Resources

Google — The Art of SLOs (Participant Handbook) — the classic SLI formula, the same source sre-and-incident-response-guide, Module 2, lesson 3 already cited.
Google — SRE Workbook, Implementing SLOs — the exact source for "the proportion of requests that were faster than some threshold," the latency SLI definition used in this lesson's SLI 2.
AWS Docs — Monitor bedrock-runtime inference using CloudWatch metrics — the source for InvocationLatency, the exact metric name this lesson's SLI 2 would need, developed with complete precision in lesson 7.
This same course, Module 1, lesson 3 — the origin of ManifestParseFailed and parse_manifest(), the complete source for this lesson's SLI 1.
This same course, Module 4, lesson 6 — the origin of validate_shipment_fields() and SHIPMENT_FIELDS_SCHEMA, the complete source for this lesson's SLI 3.

← Previous1. Module introduction: the same vocabulary, new metrics Next →3. Hands-on: CloudWatch Logs and Metrics for the extractor