Module 1: Genai In Production Vs A Notebook
2. A notebook and a production system are not the same problem
Description
cloud-security-and-guardrails-guide opened with a lesson that separated what a clean plan, a successful apply, and a green pipeline actually confirm from what a hurried reader assumes they confirm — and the conclusion was that none of that confirms the system is secure. finops-and-cost-guardrails-guide repeated the exercise with the same discipline, and reached the same conclusion structure for a different question: none of that confirms the system is cheap. This lesson does the same exercise a third time, with a different signal than the previous two — not a green pipeline, but a correct answer in Bedrock's interactive playground — and with a question neither of the two previous guides needed to ask, because neither worked with a non-deterministic component: does an answer that looks right, in a test console, confirm that the same model is secure, cheap, and reliable running in production?
The answer, again, is no. And this time the underlying reason runs deeper than in the two previous lessons: a clean plan and a correct playground answer don't fail for the same reason. A clean plan says nothing about security or cost because it never evaluated those questions — the tool does exactly what it promises, it just promises something different. A correct playground answer has an additional problem, one specific to this guide: it doesn't even guarantee that the same question, tomorrow, with the same input, produces the same answer.
Connection to the module
Lesson 1 gave you the full map and this guide's honesty commitment. This lesson installs the underlying reason why that honesty matters more here than in any previous guide: if "it works in the playground" already meant "it works in production," the rest of this guide — cost, guardrails, security, observability — would be a decorative exercise. Lesson 3, immediately after, brings this abstract idea down to Andes Cargo's real, concrete problem: the free-text manifest that process-shipment-manifest can't read.
Analogy: cooking for yourself and opening a restaurant
Cooking a new dish for yourself, on a Sunday afternoon, and deciding to open a restaurant that serves that same dish look, on the surface, like the same task: someone follows a recipe and produces food. In practice they're two problems with almost nothing in common. Cooking for yourself demands, above all, that the recipe work: that the flavor come out right, that the texture be correct, that the cook time be calibrated. If something goes wrong, you eat it anyway or you order a pizza — the cost of the mistake is yours, and it ends there. Opening a restaurant with that same dish demands solving a completely different set of problems, most of which have nothing to do with the recipe: how much does each serving cost if the main ingredient's price goes up next week? What happens if a customer has an allergy the dish doesn't disclose? Can the kitchen serve a hundred servings in one night without quality dropping by serving eighty? Who checks that today's supplier delivered the same thing as last week's? The recipe — the dish itself — is, of the hundred decisions opening a restaurant demands, probably the easiest of the hundred.
A language model answering well in Bedrock's playground is the recipe that works one Sunday afternoon. The rest of this guide — the full eight modules — is everything else opening the restaurant demands: how much it costs per serving (M2, M6), what happens if someone tries to make it say something it shouldn't (M4), whether the infrastructure serving it is declared and verified (M3, M5), whether it can handle a hundred requests without cost or quality dropping (M7), and who's responsible for it still working that way tomorrow (M8). The dish is the least of it. This entire guide is everything else.
What a correct playground answer actually confirms
When you type a test input into Bedrock's interactive console and the model responds with exactly the fields you expected — shipmentId, originCountry, destinationCountry, weightKg — that confirms, with reasonable certainty, three things: the chosen model is capable, in principle, of extracting that kind of information from that kind of text; the prompt you wrote, for that specific input, produced the expected output; and the account you're using has access enabled for that model. That's all a correct playground answer promises. It doesn't confirm that the same call, run tomorrow with the same input, produces exactly the same output — that's the nature of a language model, not a flaw in this particular call. It doesn't confirm that the call, running a hundred times a day for a month, fits within any reasonable budget. It doesn't confirm that a third party who discovers the endpoint can't manipulate the prompt to extract information it shouldn't. It doesn't confirm that the role invoking the model has permission scoped to that specific model, instead of bedrock:*. The playground evaluates the model's point-in-time capability — can this model, in principle, do this? — never operational posture — should this system, in production, work this way?
What it does NOT confirm about cost
A playground answer doesn't come with any price tag attached, and that's easy to forget when the answer arrives in seconds, apparently free. Bedrock charges per token — input and output, separately, at different prices —, not per hour the console stays open. A ten-line test in the playground doesn't tell you anything about the cost of processing ten thousand manifests a month, with variable-length text, some of which the model might need to retry. This guide's full M2 exists because this question — how much does this really cost, at scale? — has no answer without explicitly declaring a volume assumption. No tool, not the playground, not Infracost, can guess it.
What it does NOT confirm about security
A correct answer in the playground runs with the credentials of whoever is sitting at the console, with no guardrail active by default, with no one having yet tried to manipulate the prompt with a hidden instruction, with the test text containing no real sensitive information. None of those three conditions is what this same model would face in production, receiving manifests from external logistics partners, some of whom — with bad intent, or without it — might include text designed to make the model ignore its original instructions. M4 and M5 of this guide exist exactly for that gap: a managed guardrail declared as code, a custom check as defense in depth, and least-privilege IAM scoped to a specific model's ARN — none of the three exist yet when someone tests a text in the playground.
What it does NOT confirm about reliability
This is the question no previous guide in this ecosystem needed to ask with this intensity, because none worked with a fundamentally non-deterministic component. process-shipment-manifest, the function inherited from aws-core-services-guide, produces the same result for the same manifest, always — it's the guarantee that literally underpins every "What to expect (literal)" block in the previous seven guides in this ecosystem. A language model doesn't offer that guarantee: the same input text, across two different calls, can produce a slightly different extraction, even with the model's temperature set to the minimum. A correct answer in the playground, a single time, says nothing about the success rate of that same call running a thousand times. This guide's M7 builds the vocabulary — SLI, SLO, error budget — to talk about this reality with precision instead of vagueness, and the most honest metric this guide can offer without ever invoking Bedrock for real: the escalation rate from the deterministic path to the probabilistic path, a real number, calculable without a single model invocation.
The honest inventory: what "it works in the playground" still hasn't answered
| What the playground confirmed | What the playground never asked |
|---|---|
The model can extract shipmentId/originCountry/destinationCountry/weightKg from a test text | How much does it cost to process ten thousand real manifests a month, with that same model? (M2, M6) |
| The test account has access enabled for the chosen model | What role invokes the model in production, and with exactly what scope of permission? (M3, M5) |
| The answer, for that specific text, had the expected shape | What happens if the input text tries to manipulate the prompt into ignoring its instructions? (M4) |
| No one interrupted the test session with an error | Does the same call, run a thousand times, produce an acceptable success rate? (M7) |
| The test text didn't contain any real sensitive information | What happens if a real manifest carries a document number or a personal phone number? (M4) |
| The answer arrived in seconds, with no visible friction | How long does that same call take under real load, and what SLO is reasonable to promise? (M7) |
Six questions, none answered by "it works in the playground," each one paired with the exact module in this guide that resolves it — the same table structure you already saw, with different content, in the two previous guides in this ecosystem.
Why this lesson isn't a repeat, but one step further
It's worth stating, precisely, how this lesson differs from its two predecessors, not just that it imitates them. cloud-security-and-guardrails-guide M1.2 and finops-and-cost-guardrails-guide M1.3 worked on deterministic infrastructure: once Shipments is declared as PAY_PER_REQUEST and the bucket as private, those properties don't change from one execution to the next — the problem, in both cases, was that no one had asked the right question yet, not that the answer changed on its own over time. Here the problem has an additional layer: even if someone did ask the right question about security, cost, and reliability for a specific call to the model, the answer to that question can vary from one call to the next, because the model's output itself isn't fixed. This is the exact reason this guide marks, without exception, every output from a Bedrock model as "(representative)" the moment it appears — not out of a lack of rigor, but because presenting it as literal would itself be technically false.
Common mistakes
Concluding that "the playground confirms nothing" means "the playground is useless" (scope mistake). What happens: someone reads this lesson and starts distrusting the playground as an exploration step for trying out ideas. How to spot it: if your reaction is "so I shouldn't test anything there." How to fix it: the playground is still the right, fast way to confirm that a model can, in principle, handle a type of task — it's exactly the step that precedes this guide, not one this guide replaces. The point isn't that the playground is useless, it's that it confirms only one of the six questions in the table above, and the other five need the rest of this guide.
Thinking "non-deterministic" means "randomly unpredictable, with no pattern at all" (definition mistake). What happens: someone concludes that, if the model isn't deterministic, then any output is just as likely as any other, with no way to reason about it at all. How to spot it: if your conclusion is "so nothing it produces can be trusted." How to fix it: non-deterministic doesn't mean random without criteria — it means the same input can produce slightly different outputs, almost always within a reasonable quality range, not just anything. That's exactly why guardrails (M4), output schema validation (M4.6), and a smoke test harness (M7.6) exist: tools designed specifically to put reasonable limits around an output that varies, not to pretend it doesn't vary.
Looking, in this lesson, for a claim that Bedrock "is worse" than the infrastructure from previous guides (wrong-comparison mistake). What happens: someone reads the six unanswered questions and concludes Bedrock is a less mature or less reliable technology than S3 or DynamoDB. How to spot it: if your summary is "Bedrock has more problems than the rest of AWS." How to fix it: the difference isn't service quality — it's the nature of the problem. S3 and DynamoDB are deterministic by design; a language model is probabilistic by design, with the same level of engineering maturity behind it. The right question was never "is Bedrock good?" — it's "what changes in how I operate a system when one of its pieces isn't deterministic?", and that is, literally, the question the rest of this guide answers.
Exercises
Exercise 1 — Fill in the table with a seventh row of your own. This lesson's table has six rows. Based on what you already know about the Andes Cargo case, propose a seventh question "it works in the playground" also didn't answer, one that doesn't already appear in the table.
See solution
There's no single correct answer, but one solid example: "What happens if the volume of free-text manifests suddenly grows — for example, if a large logistics partner stops sending in key=value format — and the system starts escalating to the LLM far more than expected?" — the playground never answered that because it never tested more than one call at a time. This specific question is partly answered by this guide's M6 (bedrock-budget.rego, the step that stops a disproportionate volume assumption before apply). Any reasonable question about behavior at scale, not about an isolated call, counts as a valid answer.
Exercise 2 — Explain the difference between "the plan doesn't confirm security" and "the playground confirms nothing" in your own words. A colleague who already did cloud-security-and-guardrails-guide tells you: "I already know green isn't a synonym for secure, this is the same thing again." Explain how this lesson differs from that one, beyond the domain (security versus AI reliability).
See solution
A complete answer sounds, roughly, like this: "A clean plan doesn't confirm security because the tool never evaluated that question — but, once applied, the resulting resource (a bucket, a role) has a fixed security posture: private or public, it doesn't change on its own from one moment to the next. A playground answer has that same problem, plus one more: even if someone did ask the right question about that specific call, the answer can vary from one call to the next, because the model itself isn't deterministic. It's not just that 'no one asked yet' — it's that the property you'd be asking about isn't fixed over time the way a bucket's policy is."
Exercise 3 — Predict which module of this guide resolves each of the three central questions. Without looking at this lesson's table, match each of these three questions to the module in this guide that resolves it: (a) how much does this really cost, at scale?; (b) what happens if someone tries to manipulate the prompt?; (c) how often does this call fail, and how fast does it respond?
See solution
(a) Cost at scale → M2 (the Bedrock cost model, with the custom calculator) and M6 (FinOps for tokens, the budget built into the gate). (b) Prompt manipulation → M4 (Bedrock guardrails and defense in depth, including the prompt-attack detection policy). (c) Failure rate and latency → M7 (observability, latency, and evals in production, with the three SLIs of an AI workload). If you matched all three correctly, you have a clear grasp of this guide's full structure, not just its first module.
Summary and next step
In this lesson you separated what a correct answer in the Bedrock playground confirms — the model's point-in-time capability, for that specific input — from what it never asked: cost at scale, security against manipulation, reliability under repeated load. You saw the honest inventory of six unanswered questions, each one pointing to the exact module in this guide that resolves it. And you saw why this lesson, even though it repeats the "green isn't a synonym for X" structure from the two previous guides, adds a layer neither of them needed: the system's own output isn't fixed over time, so even answering the right question once isn't enough.
Before moving on you should be able to: explain, using the restaurant analogy, why "the recipe works" is the easiest part of operating a model in production; name the six questions from the honest inventory and the module that resolves each one; and explain the underlying difference between "no one asked the question yet" (the two previous guides) and "the answer can vary on its own" (this guide).
Lesson 3 brings all this theory down to Andes Cargo's real problem: the free-text manifest that process-shipment-manifest can't read, and the architecture decision that resolves it without turning the model into the main path.
Resources
cloud-security-and-guardrails-guide, Module 1, lesson 2 (02-green-does-not-mean-secure.md) — the first iteration of this pattern, on security.finops-and-cost-guardrails-guide, Module 1, lesson 3 (03-green-does-not-mean-cheap.md) — the second iteration, on cost.- AWS — Amazon Bedrock — the playground console referenced in this lesson, a conceptual starting point, not this guide's object of study.
src/paths/aws-cloud-ecosystem/STRATEGY.md— the market finding cited in lesson 1: engineers who "operate" real AI applications, not just test them, command significant market premiums.