Module 6: The Deterministic Shell
The proposal → execution pipeline
Overview
You have all the pieces of the deterministic shell, each built and measured separately: the model proposes instead of executing (L2), in a structured format (L3), from a bounded capabilities menu (L4), validated against the business rules (L5), with a core kept small (L6). This lesson puts them in order, as gates in series, in a single flow that goes from proposal to execution: structure → capability → policy → execute. And it adds the missing piece that makes all of this an operable system, not just a safe one: the audit log, which records what was proposed, what was approved, what was blocked, and at what stage.
The pipeline is the entire deterministic shell, executed end to end. Each model proposal enters through the first gate and advances while it passes; as soon as a gate rejects it, it stops there and the reason is recorded. Only the proposals that pass the three gates reach execution. You'll see the pipeline running over a batch of eight varied proposals, showing at which stage each block stopped: one at structure, two at capability, two at policy, and three that passed everything and were executed.
Connection with the module. This is the operational synthesis lesson: it assembles the gates from L3, L4, and L5 in the correct order and adds the auditing. It's the equivalent, for actions, of "the property sheet" or "the request path" from the previous modules: the complete pattern in an artifact you can take with you. The boundary with module 7 (the data loop) appears here in seed form: the audit log this lesson builds to operate the shell is also the raw material of module 7's feedback loop —each recorded decision is a datum about how the model behaves—. The boundary with AI Engineering holds: the pipeline contains and audits the actions; how the model generates them is AI Eng.
An analogy: the airport security control
Think about how you go from an airport's entrance to the boarding gate. It's not a single control: it's a sequence of gates in order, and each one verifies something different. First, the document: is your boarding pass valid and legible? If you don't have a pass, you don't pass —and it doesn't even matter where you're going—. Second, the identity: does your ID match the pass? If not, you stop there. Third, the security inspection: does what you're carrying meet the rules —nothing prohibited, liquids within their limit—? If something fails, they stop you at that point. Only if you pass the three do you get to board. And each gate verifies something the previous ones don't: the document doesn't look at your bag, the inspection doesn't check your pass.
Notice two properties of this sequence. First, the order matters: the security inspection makes no sense before verifying you have a valid pass —why inspect the bag of someone who isn't even going to fly?—. The gates go from the most basic (do you have a document?) to the most specific (does your luggage meet the rules?). Second, there's a record: each control leaves proof that you passed (or that they stopped you and why). If something goes wrong, or if someone asks "how did this person get to the gate?", there's a trace of which gates they passed and which they didn't.
That security control is the proposal → execution pipeline. The document gate is the structure check (is it a well-formed command?); the identity one is the capability one (can this agent propose this action?); the inspection one is the business-rules one (does the action meet the policy?). The order goes from the basic to the specific, just like at the airport. And the record of each control is the audit log: the trace of what was proposed, what passed, and what was stopped, so you can operate the system and answer when someone asks how an action was executed (or not).
Worked example: the complete pipeline, with auditing
Let's run the entire security control. We define the three gates as functions —stage_structure, stage_capability, stage_policy— and put them in a list, in order. Each proposal passes through the gates in sequence; as soon as one rejects it, it stops there, the stage and reason are recorded, and it doesn't advance. The ones that pass the three are executed. So the separation of gates is visible, the menu of known commands (ACTION_MENU) includes actions that exist in the platform but that this agent does not have granted (issue_store_credit, change_price): that way one of them stops at the capability gate, not at the structure one. We run a batch of eight proposals.
# Module 6, Lesson 7: the proposal -> execution pipeline.
# Assembles the three gates in order: structure (channel) -> capability
# (menu) -> business rules (policy) -> execute. Each LLM proposal passes
# through the pipeline and lands in an audit log. No network, no API.
ORDERS = {
"A-1001": {"total": 50.00, "days_since_delivery": 3, "refunded": False},
"A-1002": {"total": 120.00, "days_since_delivery": 45, "refunded": False},
"A-1005": {"total": 75.00, "days_since_delivery": 8, "refunded": False},
}
MAX_REFUND = 100.00
RETURN_WINDOW_DAYS = 30
# Menu of known commands (structure) and their required fields.
ACTION_MENU = {
"refund": {"order_id", "amount"},
"escalate_to_human": {"reason"},
"send_message": {"text"},
"issue_store_credit": {"amount"},
"change_price": {"product_id", "price"},
}
# Capabilities GRANTED to the support agent (subset of the menu).
GRANTED = {"refund", "escalate_to_human", "send_message"}
def stage_structure(p):
if not isinstance(p, dict):
return (False, "free text, not a command")
action = p.get("action")
if action not in ACTION_MENU:
return (False, f"unknown action: {action!r}")
if ACTION_MENU[action] - p.keys():
return (False, "missing required fields")
return (True, "")
def stage_capability(p):
if p["action"] not in GRANTED:
return (False, f"{p['action']} outside the agent's capabilities")
return (True, "")
def stage_policy(p):
if p["action"] != "refund":
return (True, "") # only refund touches money and needs policy
oid, amount = p["order_id"], p["amount"]
if oid not in ORDERS:
return (False, f"order {oid} does not exist")
o = ORDERS[oid]
if o["refunded"]:
return (False, f"order {oid} already refunded")
if o["days_since_delivery"] > RETURN_WINDOW_DAYS:
return (False, "outside the return window")
if amount > o["total"] or amount > MAX_REFUND:
return (False, "amount out of limit")
return (True, "")
def execute(p):
if p["action"] == "refund":
ORDERS[p["order_id"]]["refunded"] = True
return p["amount"]
return 0.0
PIPELINE = [("STRUCTURE", stage_structure),
("CAPABILITY", stage_capability),
("POLICY", stage_policy)]
PROPOSALS = [
{"action": "refund", "order_id": "A-1001", "amount": 50.00},
"I'll give you your money back right away",
{"action": "issue_store_credit", "amount": 200.00},
{"action": "change_price", "product_id": "P-1", "price": 0.01},
{"action": "refund", "order_id": "A-1002", "amount": 120.00},
{"action": "refund", "order_id": "A-9999", "amount": 40.00},
{"action": "escalate_to_human", "reason": "customer asks for a supervisor"},
{"action": "refund", "order_id": "A-1005", "amount": 75.00},
]
print(f"{'#':<3}{'action':<20}{'result':<14}{'stopped at':<12}reason")
print("-" * 92)
executed = 0
money = 0.0
stopped = {"STRUCTURE": 0, "CAPABILITY": 0, "POLICY": 0}
for i, p in enumerate(PROPOSALS):
action_name = p["action"] if isinstance(p, dict) and "action" in p else "(free text)"
blocked_at = None
reason = ""
for stage_name, stage_fn in PIPELINE:
ok, why = stage_fn(p)
if not ok:
blocked_at = stage_name
reason = why
stopped[stage_name] += 1
break
if blocked_at is None:
money += execute(p)
executed += 1
print(f"{i:<3}{action_name:<20}{'EXECUTED':<14}{'-':<12}{reason}")
else:
print(f"{i:<3}{action_name:<20}{'BLOCKED':<14}{blocked_at:<12}{reason}")
print()
print(f"Executed : {executed}/{len(PROPOSALS)} money moved: {money:.2f}")
print(f"Blocked by stage -> STRUCTURE: {stopped['STRUCTURE']} "
f"CAPABILITY: {stopped['CAPABILITY']} POLICY: {stopped['POLICY']}")What to expect. When you run the file, the output is exactly this:
# action result stopped at reason
--------------------------------------------------------------------------------------------
0 refund EXECUTED -
1 (free text) BLOCKED STRUCTURE free text, not a command
2 issue_store_credit BLOCKED CAPABILITY issue_store_credit outside the agent's capabilities
3 change_price BLOCKED CAPABILITY change_price outside the agent's capabilities
4 refund BLOCKED POLICY outside the return window
5 refund BLOCKED POLICY order A-9999 does not exist
6 escalate_to_human EXECUTED -
7 refund EXECUTED -
Executed : 3/8 money moved: 125.00
Blocked by stage -> STRUCTURE: 1 CAPABILITY: 2 POLICY: 2
Read the audit log row by row, because each block fell in the gate that corresponds to it.
Three proposals passed the three gates and were executed. The 0 (refund A-1001 $50), the 6 (escalate_to_human), and the 7 (refund A-1005 $75): each is a well-formed command, within the agent's capabilities, that meets the policy (or that, like the escalation, doesn't touch money and doesn't need policy). Only these three moved the system —$125 in legitimate refunds and an escalation to a human—. The three passed through the three gates in order, like the passenger who shows their pass, their identity, and their luggage and gets to board.
Each block fell in its gate, and that's pure diagnosis. Look at the "stopped at" column:
- The 1 (free text) stopped at STRUCTURE: it's not even a command, so it makes no sense to ask about its capability or its policy. It was rejected at the first gate, the most basic one.
- The 2 and 3 (
issue_store_credit,change_price) stopped at CAPABILITY: they're well-formed commands —they passed structure— but aren't among the capabilities granted to the support agent. Notice the important thing: they're actions that exist in the platform (they're inACTION_MENU, that's why they passed structure), but this agent doesn't have them granted (they're not inGRANTED). They're rejected at the second gate, by agent identity, not by format. - The 4 and 5 (
refundA-1002 out of window,refundA-9999 nonexistent) stopped at POLICY: they're well-formedrefunds, within the agent's capabilities, but they violate the business rules. They're rejected at the third and last gate, by policy.
The order of the gates avoids useless work. Notice that no proposal was evaluated at a gate later than the one that stopped it. The free text in row 1 never reached the policy validation —why ask about the return window of something that isn't even a command?—. It's the same logic as the airport: you don't inspect the bag of someone with no pass. The gates go from the basic (is it a command?) to the specific (does it meet the policy?), and each proposal advances only up to where it fails. This isn't just efficiency: it's diagnostic clarity —the stage where it stopped is the type of problem it has—.
The per-stage summary is an operations dashboard. The last line —"STRUCTURE: 1, CAPABILITY: 2, POLICY: 2"— isn't decoration: it's operational information. If one day that count changes abruptly —suddenly many blocks at CAPABILITY—, it tells you the model started proposing actions outside its menu (a prompt change?, a manipulation attempt?, a drift?). The audit log turns the shell from a black box that "sometimes blocks" into an observable system, where you know exactly what was proposed, what was approved, what was blocked, and why. And that observability is, additionally, the first raw material of module 7's data loop.
Going deeper: the shell as an observable pipeline
The example showed the flow; it's worth understanding the properties that make this pipeline a design artifact, not just a sequence of if.
The order of the gates is from the general to the specific. Structure → capability → policy isn't an arbitrary order: it's from the most basic question to the most specific. Structure asks "is this even a command?" —the most general, and a precondition of everything else—. Capability asks "is it a command of this agent?" —more specific, but still independent of the data—. Policy asks "is it an action that meets the rules right now?" —the most specific, and what requires consulting the source of truth—. Putting the gates in this order means each proposal is rejected at the earliest (and cheapest) stage possible, and that the expensive stages —consulting the database for the policy— only run for proposals that already passed the cheap ones. It's the same "fail fast and cheap" principle that governs any validation pipeline.
The pipeline: gates IN ORDER, from general to specific
LLM proposal
│
▼
┌──────────────┐ fails → BLOCK (logs: STRUCTURE) ────────┐
│ STRUCTURE │ is it a menu command, with fields? │
└──────────────┘ │
│ pass │
▼ │
┌──────────────┐ fails → BLOCK (logs: CAPABILITY) ───────┤
│ CAPABILITY │ is it a command of THIS agent? │
└──────────────┘ ├──► AUDIT
│ pass │ LOG
▼ │
┌──────────────┐ fails → BLOCK (logs: POLICY) ───────────┤
│ POLICY │ does it meet the business rules? │
└──────────────┘ │
│ pass │
▼ │
┌──────────────┐ │
│ EXECUTE │ money/state moves ONLY here ────────────┘
└──────────────┘ (and it's also logged)
The audit log is part of the design, not an extra. It's tempting to see the logging as something to be added "later, if there's time". In a shell that controls actions on money, it's the opposite: the auditing is a design requirement. For three reasons. Accountability: when someone asks "why was this order refunded (or not)?", the log has the exact answer —what was proposed and which gate decided—. Observability: the per-stage count is a dashboard that reveals changes in the model's behavior before they become a problem. Debugging: when something goes wrong, the log says at which gate and why, without having to reconstruct the flow. A shell without auditing contains the actions but is opaque; with auditing, it contains and explains. And explaining is what lets you operate the system, not just have it running.
The pipeline is the action shell's property sheet. If module 4 had its "guardrail stack" and module 1 its "component property sheet", this pipeline is the equivalent artifact for actions: the ordered list of gates every proposed action must pass, plus the record of what happened. When you design the shell of an AI feature that takes actions, this is the mold: what is its structure check?, what is its capabilities menu?, what are its policy rules?, how is it audited? Answering those four questions is designing the shell. The pipeline isn't one implementation among many; it's the canonical form of action containment.
Executing is just one more stage, the last. Notice a detail of the diagram: execute is at the same level as the gates, as the final stage of the pipeline, and the money moves only there, after the three validations. This materializes the whole module's thesis: the execution isn't coupled to the model's output (that was the L2 antipattern); it's at the end of a pipeline of gates the proposal had to pass entirely. The model proposes at the beginning; the system disposes at each gate; the effect happens only at the end, and only for what passed everything. "The model proposes, the system disposes" isn't a phrase: it's the structure of this pipeline.
Common mistakes
Putting the gates out of order or skipping one. What happens: the pipeline validates the policy before verifying the structure, or skips the capability gate trusting that the rules will catch everything. The result is rare errors —trying to read the order_id of something that isn't a command and crashing— or holes —an action outside the agent's menu that passes because no one checked its capabilities—. Why it happens: the order seems like a detail, and each gate seems redundant with the others. How to detect it: does your pipeline evaluate structure first, then capability, then policy, and stop the proposal at the first that fails? If not, you have disorder or holes. How to fix it: the gates go from the general to the specific, in order, and each proposal stops at the first that rejects it. Each gate protects against something the others don't see; none is superfluous.
Executing and auditing as separate steps that can desync. What happens: the code executes the action in one place and writes the log in another, and when something fails in between, the system can execute without recording (or record without executing), leaving a trace that doesn't match reality. Why it happens: the logging is treated as an optional side effect, not as part of the transaction. How to detect it: can your system move money without leaving a record, or leave a record of something that didn't happen? How to fix it: treat the execution and its record as a unit —what's executed is audited, always, in the same flow—. The log has to be a faithful mirror of what the system did, or it's no use for accountability.
Treating the log as noise and not looking at it. What happens: the pipeline audits everything, but no one looks at the log nor the per-stage count, so a change in the model's behavior —suddenly many blocks at capability, or at policy— goes unnoticed until it becomes an incident. Why it happens: the log is seen as a file for "when something fails", not as a dashboard to watch. How to detect it: if no one reviews your shell's metrics regularly, the auditing only serves as an autopsy, not as an early alert. How to fix it: the per-stage count is a live signal —watch it—. A jump in the blocks of a stage is information about the model, about an attack, or about a drift, and it's exactly the kind of signal that module 7's data loop turns into a system improvement.
Exercises
Exercise 1 — Predict the stage. Without running the code, for each of these proposals say at which gate it would stop (or if it would be executed), given the example's agent (GRANTED = {refund, escalate_to_human, send_message}, ACTION_MENU also includes issue_store_credit and change_price): (a) {action: "send_message", text: "Done"}; (b) {action: "change_price", product_id: "P-2", price: 5.00}; (c) {action: "refund", order_id: "A-1005", amount: 90.00}; (d) {action: "cancel_order", order_id: "A-1001"}.
See solution
- (a)
send_message→ EXECUTES. Structure: it's a menu command with itstextfield — passes. Capability:send_messageis inGRANTED— passes. Policy: it's notrefund, so it doesn't need rule validation — passes. The three gates: it's executed. - (b)
change_price→ BLOCKS at CAPABILITY. Structure:change_priceis inACTION_MENUwith itsproduct_idandpricefields — passes structure. Capability:change_priceis not inGRANTED(the support agent doesn't have it granted) — it stops here. It's a valid platform command, but not this agent's. - (c)
refundA-1005 $90 → EXECUTES. Structure:refundwithorder_idandamount— passes. Capability:refundis inGRANTED— passes. Policy: A-1005 exists, not refunded, 8 days ≤ 30, $90 ≤ total ($75)... wait —$90 > $75 (the order total)—, so it BLOCKS at POLICY by "amount out of limit". (Correction: the amount exceeds the order total, even though it's under the $100 limit.) Lesson: you have to check the order's real data, not just the agent's limit. - (d)
cancel_order→ BLOCKS at STRUCTURE.cancel_orderis not inACTION_MENUat all —it's not a known platform command—, so it stops at the first gate by "unknown action". It doesn't even reach capability. It's an invented/hallucinated action.
The pattern: the type of problem determines the gate. Invented action → structure; real action but not this agent's → capability; agent action that violates the rules → policy. (Note: (c) shows that "under the agent's limit" isn't enough if it exceeds the order total; both amount rules apply.)
Exercise 2 — Why the order. Explain why the pipeline puts STRUCTURE before POLICY, and what would happen if the order were reversed and the policy were validated first. Give a concrete example from the batch where the reversed order would cause a problem.
See solution
The pipeline puts STRUCTURE before POLICY because the policy validation depends on the proposal already being a well-formed command: to ask "does the amount exceed the limit?" or "does the order exist?", there first has to be an amount and an order_id to look up. The policy reasons over the action's fields; if the proposal isn't even a command (it's free text) or is missing fields, there's nothing to reason over.
If the order were reversed and the policy were validated first, row 1 of the batch —"I'll give you your money back right away" (free text)— would cause a concrete problem: stage_policy would try to read p["order_id"] and p["amount"] of a string, which would throw a TypeError (a string can't be indexed with keys) and the pipeline would crash, instead of cleanly rejecting the proposal with "not a command". The structure gate exists precisely to guarantee that, when the proposal reaches policy, it's already a command with its fields —so the policy can assume it can read them without fear—. The general → specific order isn't aesthetic: each gate prepares the ground for the next, verifying the preconditions the next takes for granted.
Exercise 3 — Design the auditing. Mercado's operations team wants to be able to answer three questions about the support agent: (1) how much money did the agent refund today?, (2) is the model proposing more actions outside its menu than last week?, (3) when a customer complains "the agent wouldn't refund me", why was it blocked? For each question, say which datum from the audit log answers it and why the auditing is part of the shell's design, not an extra.
See solution
- (1) How much did it refund today? → the record of the executed actions with their amount (the example's
moneyvariable, accumulated per day). The log of what was approved and executed gives the exact sum of money moved, without having to reconstruct it from the database. - (2) More actions outside the menu than last week? → the per-stage block count, specifically those of CAPABILITY, compared between weeks. An increase in the capability blocks means the model is proposing more actions outside its menu —a possible prompt change, manipulation attempt, or drift—. The per-stage count is a trend dashboard.
- (3) Why was this customer's refund blocked? → the log row of that proposal: the stage where it stopped and the reason ("outside the return window", "order does not exist", "amount out of limit"). The log gives the exact and auditable answer, not a guess.
Why the auditing is part of the design and not an extra: without it, the shell contains the actions but is an opaque box —it blocks and executes, but you can't say why, nor watch how it changes, nor be accountable—. The three operations questions are real needs of a system that moves money, and none can be answered without the record. A shell that controls money has to be auditable by design: accountability, observability, and debugging aren't optional features, they're requirements of operating a system that makes decisions about people's money. Additionally, that same log is the raw material of module 7's data loop: each recorded decision is a datum about the model's behavior that can improve the system.
Summary and next step
In this lesson you assembled all the module's pieces into a single flow: the proposal → execution pipeline. The gates go in order, from the general to the specific —structure → capability → policy → execute—, and each proposal stops at the first that rejects it. You saw it run over eight proposals: three passed the three gates and were executed ($125 moved, one escalation); five were blocked, each at the gate that corresponds to it —one at structure, two at capability, two at policy—. You learned that the general → specific order makes each proposal be rejected at the earliest and cheapest stage (and that each gate prepares the preconditions of the next); that the audit log is part of the design and not an extra (accountability, observability, debugging); and that executing is just the last stage, where the money moves only for what passed everything. The deterministic shell, whole and observable.
Before moving on you should be able to: order the gates from the general to the specific and justify the order; explain why each gate prepares the next; argue why the auditing is a design requirement of a shell that moves money; and read the per-stage count as an operations dashboard.
Lesson 8 is the module's capstone: you build the complete deterministic shell of Mercado's support agent. You'll take this lesson's pipeline and run it over a batch of twelve model proposals —legitimate, hallucinated, out of policy, absurd amounts, ungranted capabilities— to measure the containment rate and the money the shell protected. The deliverable brings everything together: the shell diagram, the executed code, an ADR that records the decision to contain the model's actions, and the justification of why, in this design, the LLM never touches the money directly.
Resources
- Anthropic, "Building Effective Agents" (2024) — anthropic.com/engineering/building-effective-agents. Its description of the controls and checks around an agent's actions is the basis of this lesson's gate pipeline. In English.
- Martin Fowler and Bharani Subramaniam, "Emerging Patterns in Building GenAI Apps" — martinfowler.com/articles/gen-ai-patterns. The composed-guardrails and GenAI-app-observability patterns place this audited pipeline in the complete map. In English.
- Chip Huyen, AI Engineering (O'Reilly, 2024). Its chapters on observability and monitoring of model applications treat the logging of decisions —what was proposed, what was executed— as part of the design, not as an extra. In English.
- The
observability-and-operations-guideor ecosystem equivalent, for the mechanics of logging, traces, and metrics in depth. Here we apply observability to an AI component's action pipeline; its complete discipline lives in the operations guide. In Spanish.