The LLM is a component, not the system

Overview

There's a powerful temptation when you discover how capable an LLM is: let it do everything. Let it read the customer's ticket, decide the refund, and execute it. Let it interpret the search, choose the products, and assemble the page. Let the model be the system, because it's so smart that surrounding it with layers seems like a waste. This lesson dismantles that temptation with an architectural distinction that holds up the whole guide: the LLM is a component that lives behind a boundary, not the whole system. Its job is to propose. The job of disposing —deciding which of that proposal actually executes, over money, over state, over the user's trust— belongs to a deterministic layer that surrounds it.

In lesson 2 you wrote satisfies_contract, the function that verifies the properties of an output. Here you see where that function lives and why its location is everything: it's not inside the model, it's at the boundary between the model and the system. The model pushes a proposal against that boundary; the boundary examines it against the business rules; and only what passes the exam touches the system. You're going to see, executed, the antipattern where there's no boundary —the model executes its own output and "refunds" $9999 hallucinated— against the pattern where Mercado's deterministic shell validates each proposal and blocks three out of four.

Connection with the module. Lesson 1 said "the LLM proposes, the shell disposes"; lesson 2 showed how an output is verified; this lesson places that verification in the architecture: the LLM as a bounded component behind a boundary. It's the first time you see the complete form —a core that proposes, a shell that disposes— that lesson 5 will name as the central metaphor and module 6 will develop in depth. The boundary with AI Engineering is especially important here and must be said loudly: we are not going to build the support agent. How the agent's reasoning loop is designed, how it's given tools, how its prompt is written —that's AI Engineering—. Here we only address its architectural property: that it's a component that proposes, placed behind a boundary that validates, and that it never touches the safe directly. The shell in depth (more rules, more containment patterns) is module 6; the input guardrails and the trust boundary of prompt injection are module 4.

An analogy: the teller and the manager

You walk into a bank to ask for a refund of a charge you consider an error.

The bad scenario: the teller decides and executes. The teller listens to you, finds you convincing, opens the drawer, and hands you the money. Without checking whether the charge existed, without reviewing the policy, without anyone else approving it. It's lightning-fast and very friendly. It's also a disaster waiting to happen: one persuasive —or lying— customer is enough to empty the drawer. The teller is brilliant dealing with people, but shouldn't have the authority to dispose of money on their own.

The good scenario: the teller proposes, the bank's system disposes. The teller listens to you, understands your case, and proposes a refund in the system: "customer X, charge Y, amount Z." At that point a layer the teller doesn't control steps in: the system verifies that charge Y exists, that it's within the refund window, that amount Z doesn't exceed the original charge nor the policy limit. If everything checks out, it executes. If not, it's blocked —and the teller, however convinced, can't override it—. The teller contributes what they do well (understanding the person, the nuance, the case); the system contributes what it does well (applying hard rules, with no exception, over money).

Here's the point: an LLM is the teller, not the bank. It's excellent at understanding language, nuance, intent; and that's exactly why you want to use it facing the customer. But giving it the authority to dispose —execute the refund, move the state, touch the money— is the bad scenario: a persuadable and fallible component with the keys to the drawer. The right design is the good scenario: the LLM proposes, a deterministic layer disposes. In Mercado, the support agent is the teller; the refund policy encoded in rules is the bank's system; and the boundary between the two is what keeps a hallucination —or a malicious customer— from emptying the drawer.

Worked example: the shell blocks the hallucinated proposal

We're going to model the two scenarios in code. The ai_component is again a stub: it simulates a support LLM that reads a ticket and proposes a refund. We deliberately make it propose a realistic mix —sometimes right, sometimes an order that isn't refundable, sometimes a hallucinated amount, sometimes an order that doesn't even exist—, because the point is to see what the architecture does with each case. The authoritative state (the orders, the policy) lives in deterministic structures that the model never touches.

# Lesson 3: the LLM as a COMPONENT behind a boundary, not as the system.
# The model PROPOSES; a deterministic layer (the shell) DISPOSES.
# Boundary note: here we do NOT build the support agent (that's AI Eng);
# we only show the architectural PROPERTY of placing it behind a boundary.
import random

_RNG = random.Random(7)

# System state (deterministic, authoritative). The LLM NEVER touches it.
ORDERS = {
    "A-100": {"total": 45.00, "refundable": True},
    "A-101": {"total": 12.50, "refundable": False},  # already out of window
}
REFUND_POLICY_MAX = 50.00


# --- The AI component, SIMULATED. It only PROPOSES an action. ---
def ai_component(ticket):
    # SIMULATES a support LLM that reads a ticket and proposes a refund.
    # Sometimes it proposes well; sometimes it hallucinates an amount or a nonexistent order.
    proposals = [
        {"order_id": "A-100", "amount": 45.00},   # correct
        {"order_id": "A-101", "amount": 12.50},   # order NOT refundable
        {"order_id": "A-100", "amount": 9999.00}, # hallucinated amount
        {"order_id": "Z-999", "amount": 20.00},   # nonexistent order
    ]
    return _RNG.choice(proposals)


# --- Deterministic shell: validates the proposal BEFORE touching money. ---
def deterministic_shell(proposal):
    oid = proposal["order_id"]
    amount = proposal["amount"]
    order = ORDERS.get(oid)
    if order is None:
        return (False, f"order {oid} does not exist")
    if not order["refundable"]:
        return (False, f"order {oid} outside refund window")
    if amount > order["total"]:
        return (False, f"amount {amount} exceeds the total {order['total']}")
    if amount > REFUND_POLICY_MAX:
        return (False, f"amount {amount} exceeds the policy maximum {REFUND_POLICY_MAX}")
    return (True, f"refund of {amount} for {oid} APPROVED")


# --- Antipattern: the LLM IS the system (executes its output directly). ---
def llm_is_the_system(proposal):
    # No boundary: whatever the model says gets executed. Dangerous.
    return f"EXECUTED without validation: refund {proposal['amount']} -> {proposal['order_id']}"


print("=== Antipattern: the LLM executes its own output (no shell) ===")
_RNG.seed(7)
for _ in range(4):
    p = ai_component("customer requests refund")
    print(" ", llm_is_the_system(p))

print()
print("=== Pattern: the LLM proposes, the deterministic shell disposes ===")
_RNG.seed(7)
approved = blocked = 0
for _ in range(4):
    p = ai_component("customer requests refund")
    ok, reason = deterministic_shell(p)
    tag = "APPROVED" if ok else "BLOCKED"
    if ok:
        approved += 1
    else:
        blocked += 1
    print(f"  proposal {p['order_id']:<6} {p['amount']:>8.2f}  -> {tag}: {reason}")

print()
print(f"Summary: {approved} approved, {blocked} blocked by the shell.")
print("Not a single peso left the system without passing through deterministic rules.")

What to expect. When you run the file, the output is exactly this:

=== Antipattern: the LLM executes its own output (no shell) ===
  EXECUTED without validation: refund 9999.0 -> A-100
  EXECUTED without validation: refund 12.5 -> A-101
  EXECUTED without validation: refund 20.0 -> Z-999
  EXECUTED without validation: refund 45.0 -> A-100

=== Pattern: the LLM proposes, the deterministic shell disposes ===
  proposal A-100   9999.00  -> BLOCKED: amount 9999.0 exceeds the total 45.0
  proposal A-101     12.50  -> BLOCKED: order A-101 outside refund window
  proposal Z-999     20.00  -> BLOCKED: order Z-999 does not exist
  proposal A-100     45.00  -> APPROVED: refund of 45.0 for A-100 APPROVED

Summary: 1 approved, 3 blocked by the shell.
Not a single peso left the system without passing through deterministic rules.

Read the two sections in contrast, because that's where the whole lesson is.

In the antipattern, the LLM is the system: whatever it proposes gets executed. And look at what was executed: a refund of $9999 on a $45 order (an amount hallucination), a refund on an order that was not refundable, and a refund on order Z-999 that doesn't even exist. Three of the four operations that went out would have been real losses or errors. The model isn't "bad" —it does what a probabilistic model does, proposing with confidence things that are sometimes wrong—; what's bad is the boundary-less architecture, which turned every wrong proposal into an executed action.

In the pattern, the same LLM proposes exactly the same things —the stub uses the same seed—, but now each proposal hits the deterministic_shell before touching money. The $9999 is blocked for exceeding the total; the non-refundable order is blocked by policy; the Z-999 is blocked because it doesn't exist. Only the legitimate proposal —$45 on A-100, within policy— is approved. One approved, three blocked. The summary says it literally: not a single peso left the system without passing through deterministic rules.

Notice what didn't change between the two scenarios: the model. It's identical, equally fallible in both. The only thing that changed is that in the second one there's a boundary. That's the lesson's thesis made a number: the safety of an AI system doesn't come from the model being perfect (it never will be), it comes from where you place it and what separates it from the money and the state.

Going deeper: the "propose, don't dispose" property

It's worth making the anatomy explicit, because it's a pattern you're going to apply to every AI component from here on.

        user input (untrusted)
                 │
                 ▼
       ┌───────────────────┐
       │   ai_component    │   probabilistic CORE
       │  (the LLM: stub)  │   → only PROPOSES. Touches neither state nor money.
       └───────────────────┘
                 │  proposal {order_id, amount}
                 ▼
       ┌───────────────────┐
       │ deterministic_    │   deterministic BOUNDARY / SHELL
       │     shell         │   → validates against business rules.
       └───────────────────┘
            │            │
       APPROVED       BLOCKED
            │            │
            ▼            ▼
      state / money   (nothing executes)

The LLM never has write authority. It's the hard rule. The model produces a proposal —a data structure describing a desired action—, it doesn't execute the action. Between the proposal and the execution there's always a deterministic layer that can say no. Compare it with operating-system permissions: the model runs in "user mode," with no privileges over money or state; the shell runs in "kernel mode" and is the only one that can effect changes, and only after validating. If in your design the LLM's output reaches something that writes directly —a call to the payments API, an UPDATE to the database, an email that gets sent—, you don't have this property, and you have the antipattern.

The shell is deterministic and testable with an exact assert. This is key and sometimes overlooked: although the system as a whole has a non-deterministic component, the boundary that contains it is 100% deterministic. deterministic_shell(proposal) with the same proposal always gives the same verdict. That means the most important part for safety —the one that decides what touches the money— you can test with the good old exact-equality tests: assert deterministic_shell({"order_id": "A-100", "amount": 9999.0})[0] is False. The probabilistic core is uncertain; the boundary that contains it is certain. You design so that the uncertainty lives in a small, bounded place, surrounded by verifiable certainty.

"Component" implies replaceable and bounded. Treating the LLM as a component —not as the system— has a valuable practical consequence: if a better model arrives tomorrow (from Haiku to Sonnet, from one version to another, or even a non-LLM approach for a simple case), you swap it behind the same boundary without touching the rest of the system. The shell, the business rules, the proposal contract —all of that stays—. If instead the model is the system, interwoven with everything, changing it means redoing everything. Placing the LLM as a component behind a boundary doesn't just make it safe; it makes it substitutable, which is one of the most valuable things architecture can give you against a fast-changing technology.

Where this lesson ends and module 6 begins. Here we show the shape of the pattern with a minimal shell: a few refund-policy rules. Module 6 develops it in depth —which rules, how to keep the core small, how to compose several validations, the complete containment pattern—. And module 4 adds the other half of the boundary: not just validating the model's output (what we did here), but the input —because the customer's ticket the model reads is a trust boundary, and a prompt injection there is an attack vector—. For now keep the property: propose, don't dispose; behind a boundary, not as the system.

Common mistakes

Wiring the LLM's output directly to an action with effects. What happens: to "simplify," the team wires the agent's response directly to the refunds API, or to a send_email, or to an inventory UPDATE —whatever the model says, gets done—. It works in the demo and explodes in production the day the model hallucinates or someone injects instructions. Why it happens: adding the boundary feels like unnecessary bureaucracy when the model "almost always gets it right." How to spot it: trace the path from the LLM's output to the first side effect (money, state, a message that goes out); if there's no deterministic validation in between, you have the antipattern. How to fix it: insert the shell. The model produces a proposal (data), it never executes the action. A deterministic layer validates the proposal against the business rules and is the only one that effects the change. As in the example: the hallucinated $9999 is blocked because the boundary exists.

Making the shell so "smart" it needs another LLM to validate. What happens: someone reasons "the business rules are complex, better to have another model validate the first one's proposal," and ends up with an LLM reviewing another LLM. Why it happens: "validate" is confused with "understand," and understanding is what an LLM does well. How to spot it: your validation layer is itself non-deterministic —it gives different verdicts for the same proposal—. How to fix it: the boundary must be deterministic, precisely so it's certain and testable. The rules that validate a refund (does the order exist?, is it in the window?, does the amount not exceed the total?) are hard conditions, not nuanced judgments; write them as deterministic code. If part of the judgment truly requires a model, that part is another component that proposes and needs its own deterministic boundary —you can't close the system with an infinite chain of models reviewing each other—. The certainty has to come, at some point, from deterministic rules.

Letting the LLM be the system "because it's simpler." What happens: in a prototype, the model does everything end to end and the team decides to take it to production that way because "it already works and adding layers is extra work." Why it happens: the boundary-less architecture is genuinely simpler to write, and its fragility isn't seen until there are real users, real money, and real malicious actors. How to spot it: you can't name which deterministic layer would keep a hallucination from touching the money —because it doesn't exist—. How to fix it: accept that the boundary isn't optional extra work, it's the part of the system that makes it safe. The complexity you save by removing it reappears, multiplied, as incidents. Module 5 (failures) and module 6 (shell) show how much of an AI system's robustness lives precisely in those layers around the model.

Exercises

Exercise 1 — Find the missing boundary. For each Mercado design, say whether it has the "propose, don't dispose" property or whether it's the antipattern, and if the boundary is missing, describe what deterministic layer would need to be inserted: (a) semantic search returns a list of product IDs that are passed straight to the results page; (b) the support agent generates an email and the system sends it to the customer as is; (c) "describe your product" generates a description that's published automatically without the seller seeing it; (d) the agent proposes changing a product's price.

Exercise 2 — The shell is testable. The team says: "we can't write reliable tests for the refunds agent because the LLM is non-deterministic." Explain why that claim is false, and identify exactly what part of the system can be tested with an exact assert. Write two of those asserts for the deterministic_shell from the example.

# A hallucinated proposal (amount exceeding the total) is ALWAYS blocked.
ok, reason = deterministic_shell({"order_id": "A-100", "amount": 9999.0})
assert ok is False

# A legitimate proposal within policy is ALWAYS approved.
ok, reason = deterministic_shell({"order_id": "A-100", "amount": 45.0})
assert ok is True

# A nonexistent order is ALWAYS blocked.
ok, reason = deterministic_shell({"order_id": "Z-999", "amount": 20.0})
assert ok is False

Exercise 3 — Substitutable by design. Mercado started with a large model for the support agent. Now it wants to test whether a smaller and cheaper model is enough for most tickets. Explain why, if the LLM is placed as a component behind a boundary, this change is low-risk, and what would have to be true of the design for the change not to force re-verifying the safety of the refunds.

Summary and next step

In this lesson you installed the architectural distinction that holds up the whole guide: the LLM is a component that lives behind a boundary, not the whole system. Its job is to propose; the job of disposing —deciding what touches the money, the state, the trust— belongs to a deterministic layer that surrounds it. You saw it executed: the same fallible model, with no boundary, executed $9999 hallucinated and refunds on orders that didn't exist; with the boundary, the shell blocked three out of four proposals and only let the legitimate one through. Safety didn't come from a perfect model —impossible— but from where you placed it. And you saw three consequences: the boundary is deterministic and testable with an exact assert; the component is substitutable without redoing the system; and certainty, at some point, has to come from deterministic rules, not from more models.

Before moving on you should be able to: draw the core-that-proposes / shell-that-disposes anatomy; explain why the LLM should never have direct write authority; identify in a design where the boundary is missing; and argue why the part critical for safety is still testable even though the system has a non-deterministic component.

Lesson 4 goes up a level and asks the question that closes the module's first half: if putting in an LLM is all this —a boundary, a shell, a new contract—, why do so many people believe it's "just an API call"? You're going to see, counted and executed, the five properties that arrive all at once with that "single line of code" —latency, cost, non-determinism, new failure modes, and the trust boundary— and which module of the guide each goes to. The whole iceberg under the visible line.

Resources

• Anthropic, "Building Effective Agents" (2024) — anthropic.com/engineering/building-effective-agents. The principle of keeping the AI component bounded and with limited authority, and of separating what the model proposes from what the system executes, is exactly this lesson's property. Read it for its architectural stance; how to build the agent is the boundary with AI Eng. In English.
• Martin Fowler and Bharani Subramaniam, "Emerging Patterns in Building GenAI Apps" — martinfowler.com/articles/gen-ai-patterns. The pattern of an AI component surrounded by deterministic code that validates its outputs runs through the whole article. In English.
• Chip Huyen, AI Engineering (O'Reilly, 2024). The chapters on application architecture with foundation models describe the separation between the model and the application logic that contains it —this lesson's "component, not the system"—. In English.
• Claude documentation, tool use — docs.anthropic.com. Useful to see, at a conceptual level, how a model proposes tool calls that your code decides to execute or not —the concrete mechanism behind "propose, don't dispose"—, without fixating on a model version. The logic of when to execute them is yours (the shell). In English.