7. The exact limit: why real latency and quality can't be measured here

Description

This module's lessons 3, 4, and 6 all ran real code: a structured logger, an escalation-rate calculation, a smoke test harness. This lesson documents, with the same exact honesty Module 3, lesson 6 already applied to apply, and Module 4, lesson 7 already applied to the managed guardrail's blocking, the boundary none of those three lessons can cross: extract-shipment-manifest-fields's real latency, and the semantic quality M7.5/M7.6 deliberately doesn't measure, can only be confirmed by invoking the real model — something this $0 lab, declared since Module 1, never does.

Connection to the module

This lesson revisits, for latency and quality, exactly the same pattern Module 1, lesson 7 established for list-foundation-models, Module 3, lesson 6 applied to apply, and Module 4, lesson 7 applied to the managed guardrail's blocking: document the real schema of what would be measured, precisely and with an officially verified source checked today, never presenting an invented number as if it were measured.

---

Step 1 — The real metrics Bedrock would publish, if an invocation happened

Amazon Bedrock, for any real invocation through bedrock-runtime, publishes a fixed set of metrics to CloudWatch, under the AWS/Bedrock namespace — verified today against the official documentation:

Metric	Unit	Exact definition
InvocationLatency	Milliseconds	"The time from when a request is sent to when the last token is received."
TimeToFirstToken	Milliseconds	"Time from when a request is sent to when the first token is received" — only for streaming operations (ConverseStream, InvokeModelWithResponseStream)
InputTokenCount / OutputTokenCount	Count	Number of tokens in the invocation's input / output
Invocations	Count	Number of successful invocations to Converse/ConverseStream/InvokeModel/InvokeModelWithResponseStream
InvocationThrottles	Count	Invocations the system throttled due to quota

— AWS Docs — Monitor bedrock-runtime inference using CloudWatch metrics

These six rows aren't a list invented for this lesson — they're, literally, AWS's official table's rows, transcribed unchanged. And here's this lesson's central point, put as simply as possible: each of these metrics only has a value if a real invocation produced that value. InvocationLatency isn't a number AWS publishes ahead of time, like a price on a list — it's the result of timing a specific call, which never happens in this lab.

---

Step 2 — Why a model's latency isn't a fixed number, not even in theory

It's worth understanding, with real technical precision, why latency couldn't be a single number even if this guide did invoke the model. The same AWS documentation that defined InvocationLatency explains the exact mechanics:

"An inference request passes through two compute-bound stages on the model host: Prefill. The model processes the entire input prompt in a single forward pass and produces the first output token. The duration of this stage scales primarily with input length and is the main driver of TimeToFirstToken. Decode. The model generates each subsequent output token sequentially, one token per forward pass. The total time of this stage scales with the number of output tokens."

— AWS Docs — Diagnose InvocationLatency increases using output tokens per second (OTPS)

And the exact relationship between the three metrics, also cited verbatim from that same source:

   InvocationLatency (ms) = TimeToFirstToken (ms) + (OutputTokenCount / OTPS) * 1000

This means InvocationLatency depends on two variables that change with every manifest: how much text the input manifest has (which determines the prefill stage, and therefore TimeToFirstToken), and how many tokens the response produces (which determines the decode stage). A three-line free-text manifest and a three-paragraph one, with the same model, at the same moment, would produce different InvocationLatency values — not due to service inconsistency, but because the physics of the calculation (more input text, more output text) determines it that way. There is no, nor could there be, a single "latency number" for Nova Lite — only a distribution of values, over real traffic, with the exact shape the formula above describes.

   WHY THERE'S NO FIXED LATENCY NUMBER, NOT EVEN IN THEORY

   Short manifest, short response          Long manifest, long response
        │                                       │
        ▼                                       ▼
   Fast prefill -> low TimeToFirstToken     Slow prefill -> high TimeToFirstToken
        │                                       │
        ▼                                       ▼
   Fast decode (few tokens)                 Slow decode (many tokens)
        │                                       │
        ▼                                       ▼
   LOW InvocationLatency                    HIGH InvocationLatency

   Same model. Same account. Same minute. DIFFERENT values -- by design.

---

Step 3 — The honest finding: not even AWS's latency initiative covers the chosen model

Before concluding "there's no published number at all," it's worth confirming one really was precisely searched for. AWS does publish a specific initiative to reduce latency — verified today against the official announcement:

"Amazon Bedrock now offers latency-optimized inference for foundation models [...] currently supports Anthropic's Claude 3.5 Haiku model, as well as Meta's Llama 3.1 405B and 70B models."

— AWS — Introducing latency-optimized inference for foundation models in Amazon Bedrock

Nova Lite — the model GENAI-COST-PROFILE.md (Module 2, lesson 8) chose for Andes Cargo — isn't on that list. This isn't a minor detail: it means not even AWS's most direct piece of marketing about Bedrock latency — the one that would include, if it existed, a millisecond comparison — covers this guide's specific model. The only thing the Nova family's public documentation offers is a qualitative claim, with no number:

"Nova Micro delivers the lowest latency responses in the Amazon Nova family of models."

— AWS News Blog — Introducing Amazon Nova

Note, moreover, that this quote is about Nova Micro, not Nova Lite — the model Andes Cargo chose is, according to AWS's own documentation, the second-fastest in its family, not the fastest, with no number quantifying that difference. This guide searches with the same research discipline as every previous lesson — WebFetch against official documentation, today, not training memory — and the honest finding is that AWS publishes no millisecond range for Nova Lite, not on its model card, not on the launch blog, not in the latency-optimized initiative.

---

Step 4 — Representative example: the format a real InvocationLatency would have

With Step 1's real schema confirmed, and with no invented number, this is what a real query against this metric would look like, if extract-shipment-manifest-fields really invoked Nova Lite:

awslocal cloudwatch get-metric-statistics \
  --namespace AWS/Bedrock \
  --metric-name InvocationLatency \
  --dimensions Name=ModelId,Value=amazon.nova-lite-v1:0 \
  --start-time 2026-08-14T14:00:00Z \
  --end-time 2026-08-14T15:00:00Z \
  --period 3600 \
  --statistics Average p50 p99

What to expect (representative — the command's and the response's structure is verified against the API's real schema; no numeric Datapoints value ran here, nor could it, without a real invocation; and even if it did, Bedrock — "Included in Plans: Ultimate" — isn't available on this lab's Hobby plan either, the same limitation Module 1, lesson 7 and Module 3, lesson 6 already confirmed):

{
    "Label": "InvocationLatency",
    "Datapoints": [
        {
            "Timestamp": "2026-08-14T14:00:00+00:00",
            "Average": "VARIABLE -- depends on each manifest's length in that period",
            "p50": "VARIABLE",
            "p99": "VARIABLE",
            "Unit": "Milliseconds"
        }
    ]
}

Notice this example's "Average" field: it doesn't say a number, it says "VARIABLE", followed by the exact reason. This is a deliberate decision in this lesson, not a formatting limitation — any specific number there ("842", "1250", any figure) would have been, precisely, exactly the kind of invention this guide has refused to make since Module 1: a number that sounds precise, with no real measurement behind it.

---

Step 5 — Semantic quality: the same limit, the same underlying reason

This module's lesson 5 already drew the line between shape and meaning; this lesson confirms the reason semantic quality can't be measured here is the exact same reason latency can't be measured: both depend on a real invocation. Without a real Nova Lite response to a real free-text manifest, there's nothing to compare expectedFields against — the field evals/fixtures/sample_manifests.json (Module 7, lesson 6) already declares, but that lesson's harness never reads. The connection to the previous lesson is direct:

   evals/fixtures/sample_manifests.json (Module 7, lesson 6)

   expectedFields         --  what WOULD be correct (documented,
                               for human reading)
   representativeModelResponse -- what the harness DOES compare (shape,
                               executed, this lesson confirms it)

   Comparing both field by field = semantic evaluation = requires
   a REAL Bedrock response, which this guide never invokes

---

This module's complete honesty table, now closed

Piece	Schema/structure verified	Real number/data
InvocationLatency/TimeToFirstToken (definition, units, OTPS formula)	Yes — cited verbatim from AWS Docs, today	No — no real invocation occurred
Nova Lite-specific latency range	Not published by AWS, confirmed by direct search	No
expectedFields vs. representativeModelResponse (Module 7, lesson 6)	Yes — the harness compares shape, with real code	No — no semantic comparison occurs in this guide

---

Common mistakes

Looking up a latency number for Nova Lite on some third-party blog or unofficial comparison and presenting it as if it were from this lesson (filling in, on your own, what this lesson leaves without a number). What happens: someone, unsatisfied with "VARIABLE" as an answer, looks up a millisecond figure on a third-party benchmarking site and cites it as if it were part of this guide. How to spot it: if your notes on this lesson include a latency figure from a source that isn't docs.aws.amazon.com or aws.amazon.com. How to fix it: any number from a third-party site measures conditions this guide can't verify — which provider, what network configuration, what exact prompt size they used; citing it here, without that verification, would break the same source discipline every Resources section in this guide already requires. The correct honest move, if you really need a number, is to run your own measurement against your own Bedrock account — the same advice Module 3, lesson 6 already gave for apply.

Concluding that, because there's no published number, Nova Lite's latency is "bad" or "slow" (reading the absence of data as a negative signal). What happens: someone interprets "AWS doesn't publish a specific number for Nova Lite" as evidence the model is slow. How to spot it: if your takeaway from this lesson is "Nova Lite must be slow because it's not on the latency-optimized list." How to fix it: the absence of a published number is no evidence at all about real performance — it means, exclusively, that AWS didn't include that specific model in that particular marketing initiative, for reasons this lesson can't know. GENAI-COST-PROFILE.md (Module 2, lesson 8) was already explicit that choosing Nova Lite is "a judgment call, not a measurement" — the same honesty applies here: neither good nor bad, simply not measured.

Presenting Step 2's diagram (prefill/decode) as if it were evidence a real invocation ran (confusing explaining the mechanics with having measured them). What happens: someone, excited about the InvocationLatency = TimeToFirstToken + (OutputTokenCount / OTPS) * 1000 formula's technical precision, presents it as if it came from their own data. How to spot it: if your description of this lesson says "we measured that latency depends on..." instead of "AWS's documentation explains that...". How to fix it: the formula and the two-stage explanation are, precisely, verbatim quotes from AWS's official documentation — the same "schema verified, content not verified" discipline Module 4, lesson 7 already applied to ApplyGuardrail — they explain why latency varies, without this guide ever having measured, at any point, how much it varies in practice.

---

Exercises

Exercise 1 — Using Step 2's formula, calculate what would happen to InvocationLatency if TimeToFirstToken stays fixed at 400ms, but OutputTokenCount rises from 150 to 300 tokens (double), with a constant OTPS of 55 tokens/second (the example value AWS's official documentation uses). How much does InvocationLatency change?

See solution

With the formula InvocationLatency = TimeToFirstToken + (OutputTokenCount / OTPS) * 1000: with 150 output tokens, InvocationLatency = 400 + (150 / 55) * 1000 ≈ 400 + 2,727 = 3,127ms. With 300 output tokens, InvocationLatency = 400 + (300 / 55) * 1000 ≈ 400 + 5,455 = 5,855ms. Doubling the number of output tokens doesn't exactly double total InvocationLatency (because TimeToFirstToken stays fixed, not doubling), but it does almost exactly double the decode portion — from 2,727ms to 5,455ms. This exercise confirms in concrete numbers exactly what Step 2 explained in prose: the response's length — which depends on how many fields it extracts and how verbose the model is — is a direct factor in total latency, not a minor detail.

Exercise 2 — Explain why this lesson chooses to cite AWS's OTPS formula instead of simply saying "latency depends on text size," without the formula. What does this lesson gain by including the exact equation?

See solution

Including the exact equation turns a qualitative claim ("depends on size") into a verifiable quantitative relationship — the same precision discipline that distinguishes this guide from a superficial explanation at any other point. With the formula, a reader can, as Exercise 1 just demonstrated, calculate for themselves how latency would change under a specific hypothetical scenario, without relying on this lesson's word for it. Without the formula, "depends on size" is a claim that sounds reasonable but that no one can verify or use to reason about a concrete case — exactly the difference between citing an API's real schema (as Module 4, lesson 7 already did with ApplyGuardrail) and simply describing, from memory, how something "probably" works.

Exercise 3 — Predict what would happen to this lesson's honesty table's "Nova Lite-specific latency range" row if, six months from now, AWS added Nova Lite to its latency-optimized model list. Would that be enough for this guide to cite a real number?

See solution

It wouldn't be entirely enough, though it would meaningfully change the situation. AWS adding Nova Lite to the latency-optimized list would confirm the initiative exists for that model — and it would probably come with some qualitative or quantitative comparison in the corresponding announcement, as already happened for Claude 3.5 Haiku and Llama 3.1. But a millisecond range cited from a marketing announcement is still different from a real measurement, run against Andes Cargo's specific workload, with its own real manifest sizes. This guide would cite that new number, if it existed, with the same "representative" label it uses today for any figure it hasn't measured itself — the correct update would be adding the citation, not removing the label — until the day a real invocation, against Andes Cargo's account, produced its own InvocationLatency value.

---

Summary and next step

This lesson documented, precisely and with no invented number, this module's final limit: InvocationLatency/TimeToFirstToken are real, well-defined metrics, cited verbatim from AWS's official documentation, with an exact formula (OTPS) explaining why latency would never be a single number even with Bedrock really running. You confirmed, with a direct search against official sources, that not even AWS's latency-optimized initiative covers Nova Lite, the model chosen for Andes Cargo — an honest finding, not an invented limitation. And you revisited lesson 5's line: semantic quality can't be measured here, for the exact same reason latency can't be measured — both depend on a real invocation this guide never makes.

Before moving on you should be able to: name Bedrock's real latency metrics and their unit; write from memory the formula relating InvocationLatency, TimeToFirstToken, and OTPS; and explain why not even a model with "latency-optimized" marketing would guarantee a fixed number for any specific workload.

Lesson 8, this module's closing project, integrates the three SLIs — lesson 4's harness (literal), lesson 6's (calculable), and this lesson's exact limit (representative) — into a single documented dashboard, with the same honesty ledger every project in this guide has already modeled.

Resources

1. AWS Docs — Monitor bedrock-runtime inference using CloudWatch metrics — exact source for InvocationLatency/TimeToFirstToken, cited in this lesson's Steps 1 and 4.
2. AWS Docs — Diagnose InvocationLatency increases using output tokens per second (OTPS) — source for Step 2's prefill/decode formula.
3. AWS — Introducing latency-optimized inference for foundation models in Amazon Bedrock — source for Step 3's finding: Nova Lite isn't on the list of supported models.
4. AWS News Blog — Introducing Amazon Nova — source for the qualitative claim about Nova Micro, cited in Step 3.
5. This same course, Module 3, lesson 6 and Module 4, lesson 7 — the exact same "honest attempt" pattern this lesson reapplies to latency and quality.
6. This same course, Module 2, lesson 8 — GENAI-COST-PROFILE.md, the source for choosing Nova Lite as "a judgment call, not a measurement," cited in this lesson.