The deterministic shell and the data loop

Overview

The previous five lessons filled eight of the sheet's ten fields. This lesson fills the two that were missing, the ones that close the containment: deterministic_shell (M6) and feedback_loop (M7). They're the two pieces that turn the support agent from "a model with protections" into "a complete AI-native system": the deterministic shell contains what the model proposes —the model proposes, the system disposes, measured in money— and the data loop closes the circle —the observability sees the quality, and the feedback feeds back lesson 4's eval-set—.

These two pieces seem different —one protects the money, the other improves the quality— but they share the heart of the method: a deterministic layer around the probabilistic core. The shell is that layer validating actions; the data loop is that layer observing the quality and feeding it back. And there's a connection this lesson makes explicit: the same shell that blocks a dangerous refund generates the trace the data loop uses to improve the system. Containing and improving aren't two separate systems; they're two faces of the same deterministic shell.

Connection with the module. This lesson is the one that closes the capstone's cycle. The deterministic shell is the hard guarantee lessons 4 and 5 anticipated —the boundary that validates every proposal against the policy, now measured in the money it protects—. And the data loop connects back with lesson 4: the thumbs_down the observability captures live become new eval-set cases, which the next run of the eval gate will use. With this, the sheet's ten fields are filled and the system is complete —ready for lesson 8, which runs it whole—. The boundary with AI Engineering holds: here the loop is the architecture decision (setting up the feedback loop); how to train a model with the data the loop accumulates is AI Engineering.

An analogy: the manager who authorizes and the suggestion box that improves the store

Go back to the Mercado store, but look at two things a good manager does, distinct but connected.

The first: the manager authorizes the refunds. A brilliant employee serves the customers and, when someone asks for a refund, doesn't execute it themselves —they take the request to the manager—. The manager reviews it against the store's rules: does the order exist?, is it within the window?, is the amount correct? Only if it passes does the manager authorize and the money move. The employee proposes; the manager disposes. And here's the measurable value: when the employee, confused or tricked, proposes a $9999 refund for a $75 order, the manager stops it before a cent goes out. The manager doesn't make the employee smarter; they take away their power to move the money alone.

The second: the manager reads the suggestion box. Every customer who left unsatisfied —the employee didn't know how to answer their question— dropped a note in the box. The manager reads them, and each complaint becomes something concrete: a question the employee should know how to answer next time, which gets added to the training manual. The store improves because the manager closes that loop —reads the box and updates the manual—. A manager who put the box but never read it would have the illusion of improving without improving: the complaints would pile up without changing anything.

Notice the connection between the two: it's the same manager. The refund authorization (containing) and the box reading (improving) are done by the same person, with the same information. In fact, the refund request the manager authorized also tells them something about what's happening in the store —which products get returned more, which questions come up—. In the support agent, the deterministic shell is the manager who authorizes, and the data loop is the same manager reading the box. This lesson sets up both, and shows that they're the same deterministic shell with two jobs.

Worked example: the shell protects the money, the loop closes the circle

We're going to execute the two pieces in a single program. Part A measures the deterministic shell: it takes six model proposals (good and bad mixed) and compares two designs —the naive, where the model executes directly, and the shell one, where the model proposes and the shell disposes—, measuring the money the shell protects. Part B closes the data loop: the observability aggregates the live quality signal (approval_rate), and each thumbs_down becomes a new case of the module 3 eval-set.

# M8 Lesson 7 — the DETERMINISTIC SHELL (M6) contains, and the DATA LOOP (M7)
# closes the circle. Part A: the model proposes, the shell disposes, measured in
# money. Part B: observability (approval_rate) + the feedback that feeds back
# the M3 EVAL-SET. Model STUB; fixed data; no network or APIs.

# ==================================================================
# PART A — The deterministic shell contains the bad proposals (M6).
# ==================================================================
ORDERS = {
    "A-1001": {"total": 50.00,  "days_since_delivery": 3,  "refunded": False},
    "A-1002": {"total": 120.00, "days_since_delivery": 45, "refunded": False},
    "A-1003": {"total": 30.00,  "days_since_delivery": 5,  "refunded": True},
    "A-1005": {"total": 75.00,  "days_since_delivery": 8,  "refunded": False},
}
MAX_REFUND = 100.00
RETURN_WINDOW_DAYS = 30

# The same proposals from the probabilistic core: good and bad mixed.
PROPOSALS = [
    {"action": "refund", "order_id": "A-1001", "amount": 50.00},
    {"action": "refund", "order_id": "A-1002", "amount": 120.00},
    {"action": "refund", "order_id": "A-1003", "amount": 30.00},
    {"action": "refund", "order_id": "A-9999", "amount": 40.00},
    {"action": "refund", "order_id": "A-1005", "amount": 5000.00},
    {"action": "refund", "order_id": "A-1005", "amount": 75.00},
]


def deterministic_shell(p):
    oid, amount = p["order_id"], p["amount"]
    if oid not in ORDERS:
        return (False, "order does not exist")
    order = ORDERS[oid]
    if order["refunded"]:
        return (False, "already refunded")
    if order["days_since_delivery"] > RETURN_WINDOW_DAYS:
        return (False, "outside window")
    if amount > order["total"]:
        return (False, "amount > total")
    if amount > MAX_REFUND:
        return (False, "amount > limit")
    return (True, "approved")


paid_naive = sum(p["amount"] for p in PROPOSALS)            # the LLM executes directly
paid_shell = sum(p["amount"] for p in PROPOSALS if deterministic_shell(p)[0])

print("=== Part A: the deterministic shell (M6) ===")
print(f"  no shell (the LLM executes):   paid {paid_naive:.2f}  ({len(PROPOSALS)} actions)")
executed = sum(1 for p in PROPOSALS if deterministic_shell(p)[0])
print(f"  with shell (propose/dispose):  paid {paid_shell:.2f}  ({executed} actions)")
print(f"  money protected: {paid_naive - paid_shell:.2f}  "
      f"(dangerous proposals blocked before touching the money)")

# ==================================================================
# PART B — The data loop: observability + feedback -> eval-set (M7 -> M3).
# ==================================================================
# The M3 eval-set starts with these cases (the same the gate uses).
EVAL_SET = [
    {"id": "q1", "question": "where is my order",         "must_contain": "tracking"},
    {"id": "q2", "question": "how do I return a product", "must_contain": "return"},
    {"id": "q3", "question": "how long does shipping take", "must_contain": "3 to 5 days"},
]

# The live traffic: one row per resolved ticket, with tokens, cost and the
# human agent's thumbs (the quality signal). Fixed data.
LIVE_TRAFFIC = [
    # (ticket, agent_response, tokens, cost_usd, thumbs)
    ("where is my order",         "You can see the tracking in your profile.", 180, 0.0016, "up"),
    ("how do I return a product", "Go to your order and press Return.",        175, 0.0015, "up"),
    ("my gift coupon doesn't apply", "I don't have information about that.",    160, 0.0014, "down"),
    ("can I pick up in store",    "I'm not sure about that.",                  150, 0.0013, "down"),
    ("how long does shipping take", "Shipping takes 3 to 5 business days.",     170, 0.0015, "up"),
    ("how do I report a seller",  "Sorry, I can't help you with that.",        155, 0.0013, "down"),
]

ups = sum(1 for t in LIVE_TRAFFIC if t[4] == "up")
approval_rate = ups / len(LIVE_TRAFFIC)
avg_tokens = sum(t[2] for t in LIVE_TRAFFIC) / len(LIVE_TRAFFIC)
total_cost = sum(t[3] for t in LIVE_TRAFFIC)

print()
print("=== Part B: observability for AI (M7) ===")
print(f"  tickets            : {len(LIVE_TRAFFIC)}")
print(f"  approval_rate      : {approval_rate:.0%}  "
      f"({'ALERT: low quality' if approval_rate < 0.80 else 'OK'})")
print(f"  avg_tokens         : {avg_tokens:.0f}")
print(f"  total cost (live)  : ${total_cost:.4f}")

# The loop CLOSES: each thumbs_down becomes a new eval-set case.
new_cases = [t[0] for t in LIVE_TRAFFIC if t[4] == "down"]
print()
print("=== The data loop closes: feedback -> eval-set (M7 -> M3) ===")
print(f"  eval-set before : {len(EVAL_SET)} cases")
for i, question in enumerate(new_cases, start=len(EVAL_SET) + 1):
    EVAL_SET.append({"id": f"q{i}", "question": question, "must_contain": "?"})
    print(f"    + new case from feedback: {question!r}")
print(f"  eval-set after  : {len(EVAL_SET)} cases")
print()
print("The live thumbs_down became eval-set cases. By fixing them,")
print("the M3 gate will rise and the M7 flywheel turns again. The same")
print("shell that protects the money (Part A) feeds the quality (Part B).")

What to expect. When you run the file, the output is exactly this:

=== Part A: the deterministic shell (M6) ===
  no shell (the LLM executes):   paid 5315.00  (6 actions)
  with shell (propose/dispose):  paid 125.00  (2 actions)
  money protected: 5190.00  (dangerous proposals blocked before touching the money)

=== Part B: observability for AI (M7) ===
  tickets            : 6
  approval_rate      : 50%  (ALERT: low quality)
  avg_tokens         : 165
  total cost (live)  : $0.0086

=== The data loop closes: feedback -> eval-set (M7 -> M3) ===
  eval-set before : 3 cases
    + new case from feedback: "my gift coupon doesn't apply"
    + new case from feedback: 'can I pick up in store'
    + new case from feedback: 'how do I report a seller'
  eval-set after  : 6 cases

The live thumbs_down became eval-set cases. By fixing them,
the M3 gate will rise and the M7 flywheel turns again. The same
shell that protects the money (Part A) feeds the quality (Part B).

Let's walk through the two parts, because together they close the capstone's containment.

Part A — the shell protects $5,190. The six proposals include two legitimate ones (A-1001 for $50, A-1005 for $75) and four dangerous ones (an out-of-window order, a double refund, a nonexistent order, and a $5000 refund of a $75 order). In the naive design, where the model executes directly, all six refunds go out: $5,315. In the shell design, where the model proposes and the shell validates every proposal against the policy before executing, only the two legitimate ones pass: $125. The difference —$5,190 protected— is the exact value of moving the power to execute from the model to the shell. And notice the essential thing, module 6's thesis: we did nothing to improve the model —it's the same model, the same proposals—; we only took away its power to execute directly and gave it to a deterministic layer. The safety didn't come from a better model; it came from the model not executing.

Part B — the observability sees what a server log doesn't see. The observability aggregates the live quality signal: of the six tickets, three received thumbs_down, giving an approval_rate of 50% —marked ALERT: low quality—. Here's what a classic server log would miss: those six tickets all returned their response (status 200, if you measured the server), but three of the responses were bad ("I don't have information about that"). The HTTP status says the service responded; the approval_rate says it responded badly to half. Only the quality signal —which someone had to design to capture it— reveals the degradation. And the observability also aggregates the live tokens and cost, lesson 3's budget metrics watched in production.

Part B — the loop closes: the feedback feeds the eval-set. And here's the connection that closes the capstone. The three thumbs_down —"my gift coupon doesn't apply", "can I pick up in store?", "how do I report a seller?"— were questions the agent didn't know how to answer. The data loop turns them into three new eval-set cases, which grows from 3 to 6 cases. The next time you run lesson 4's eval gate, those cases will be there: if the team fixes the agent to answer them, the score rises; if not, the gate keeps marking them as failures. That's how module 7's flywheel spins: the use generates feedback, the feedback feeds the eval-set, a better agent passes the gate, and the system improves. The snowball picked up its first handful of snow.

And the connection between the two parts, which is the lesson's point: it's the same shell. Part A (containing) and Part B (improving) aren't two separate systems. The deterministic shell that validates every refund proposal (Part A) generates, in each validation, a trace —what the model proposed, what the shell decided, what feedback the human gave— and that trace is exactly what the data loop observes and feeds back (Part B). The manager who authorizes the refunds is the same one who reads the box. Containing and improving are two jobs of the same deterministic layer around the probabilistic core.

Going deeper: containing and improving are the same shell

Validating without veto power doesn't count as a shell. A subtle error Part A leaves implicit and worth stating: the shell only protects if its "no" stops the execution. A system that logs "this proposal seems out of policy" while executing it anyway —"we're monitoring it"— isn't containing; it's doing an autopsy with good documentation. Monitoring an embezzlement while it happens isn't containing it. In the code, the difference is that paid_shell only sums inside the branch where deterministic_shell(p)[0] is True —the validation has veto power over the payment—. If the validation ran after the payment (as in the naive design, which would only label the damage), it would protect nothing: the $5000 would have already gone out. The validation goes before the execution and its "no" stops, or it's not a shell.

AI observability needs a signal a classic log doesn't have. Part B showed the approval_rate, and it's worth understanding why it's the metric that changes everything. A classic server log measures whether the service responded (2xx status, latency). For an AI component that's not enough: a hallucinated or irrelevant response also returns HTTP 200. AI observability adds three columns the classic log doesn't have —tokens (which are cost and latency), cost (the bill per feature), and the quality signal (the approval_rate, whether it responded well)—. The quality signal is the truly new one: without it, your AI feature is a black box that reports "I responded" without reporting "I responded well", and a feature can degrade for weeks with the dashboard green. The observability's approval_rate is the live version of what lesson 4's eval-set measures against fixed cases: the eval lets you not deploy something bad, the observability lets you detect that something good degraded in production.

The loop only spins if it closes. Part B turned the thumbs_down into eval-set cases —that's the closing—. But module 7's flywheel only compounds if the loop really closes: capturing the feedback isn't enough. A team that puts the thumbs up/down, sees the users use it, and assumes "it already has the flywheel" has a snowball on ice —it moves but doesn't grow—, because the data piles up without being fed back. The closing is what Part B does: each thumbs_down becomes an eval-set case, which the next run of the gate will use. Without that step, the observability would be a thermometer that measures without improving anything. Capturing is visible (a button appears); closing is invisible (the datum reaches the eval-set), and it's what really makes the wheel spin.

The same trace serves the containment and the improvement. This is the idea that unites the two parts and closes the method. Every time the shell validates a proposal (Part A), it produces a trace: the ticket that arrived, the model's proposal, the shell's decision (approved/blocked and why), and —if there was a human— their feedback. That trace has double value: for the containment, it's the audit record (who approved what, why a refund was blocked); for the improvement, it's the datum that feeds the eval-set and the observability. Designing the shell so it emits that trace is what makes containing and improving the same system. An architect who sets up the shell without thinking about the trace has containment without improvement; one who designs it to observe has both with the same piece.

Common mistakes

Putting the validation after executing, not before. What happens: the team does validate the refund proposals, but does it after moving the money —"we refund and then we check if it was right"—. Since the actions are irreversible, the late validation only discovers the damage; it doesn't prevent it. Why it happens: sometimes by design (it seems simpler to execute first), sometimes by a bad ordering of the operations. How to detect it: in your flow, does the if that decides whether the proposal is valid run before or after the line that executes it? If it's after, you're doing autopsies. How to fix it: the validation goes before the execution, always; the action is only executed in the branch where the validation passed. It's the difference between the example's $125 and $5,315.

Capturing feedback and not closing it. What happens: the team puts the thumbs up/down, sees the customers use it, and assumes "we already have the flywheel". But the thumbs_down pile up without becoming eval-set cases nor adjusting anything, and the feature's quality stays flat —the snowball on ice—. Why it happens: capturing is visible (a button appears) and gets confused with closing (which is invisible). How to detect it: you have weeks of saved feedback and your quality curve is flat. How to fix it: close the loop —each thumbs_down must become an eval-set case (like in Part B), and from there guide the correction—. Capturing without closing doesn't move the needle; the flywheel spins only when the loop closes.

Monitoring like a classic server, with no quality signal. What happens: the team monitors the AI feature with the same stack as its classic services —status, latency, error rate— and never adds a quality signal. The feature degrades (a model change, drift, a prompt someone touched) and the dashboard stays green because bad responses also return 200. The problem is discovered from customer complaints, weeks later. Why it happens: the instinct is to reuse the monitoring you already have, and that monitoring never had to measure "was the response right?". How to detect it: your AI feature's dashboard doesn't have a quality column (approval_rate or equivalent). How to fix it: add the quality signal to the dashboard —the status says whether it responded, the quality says whether it responded well, and for an AI feature they're different things—.

Exercises

Exercise 1 — Why the same model paid $5,315 and $125. The two designs in Part A use the same six model proposals. Explain why one paid $5,315 and the other $125, what element the shell design has that the naive one doesn't, and what it corresponds to in the code.

Exercise 2 — The thumbs_down that orders the priorities. In Part B, three tickets received thumbs_down. Imagine that in a month of real traffic, "my coupon doesn't apply" receives 400 thumbs_down, "pick up in store" receives 50, and "report a seller" receives 15. In what order would you fix the three, and why is the feedback volume a signal of priority and not just of what's broken?

Exercise 3 — The trace that serves the containment and the improvement. The deep dive said that the same shell trace serves to contain (audit) and to improve (feedback). Design what fields the trace the shell emits when validating a refund proposal would have, and show how each field serves one of the two uses (or both).

trace = {
    "ticket_id": "T-8842",              # the ticket that originated the proposal
    "customer_message": "...",          # the customer's message (trust boundary)
    "model": "cheap",                   # which model it was routed to (cascade, M2)
    "proposal": {"action": "refund", "order_id": "A-1005", "amount": 5000.0},
    "shell_decision": "BLOCKED",        # approved / blocked
    "shell_reason": "amount > total",   # why
    "tokens": 210, "cost_usd": 0.0018,  # observability (M2/M7)
    "human_feedback": None,             # thumbs / correction, if there was one (M7)
    "injection_flagged": False,         # the input guardrail flagged something (M4)
}

Summary and next step

In this lesson you filled the last two fields of the sheet: deterministic_shell and feedback_loop. With the manager who authorizes refunds and reads the suggestion box, you saw that containing and improving are two jobs of the same deterministic layer. And you executed it in two parts: the deterministic shell protected $5,190 —the model proposed six actions, four dangerous, and the shell let through only the two legitimate ones, without making the model better, only taking away its power to execute—; and the data loop closed the circle —the observability revealed an approval_rate of 50% a server log wouldn't see, and the three thumbs_down became new cases of the module 3 eval-set—. The connection that closes the method: it's the same shell: the one that validates the proposals emits the trace the data loop observes and feeds back. With this, the sheet's ten fields are filled and the system is complete.

Before moving on you should be able to: explain why the validation with veto power goes before executing; measure the money a shell protects; distinguish AI observability (with a quality signal) from a classic server log; and explain how the feedback closes the loop toward the eval-set and why the feedback volume prioritizes the corrections.

Lesson 8 is the capstone's deliverable and the close of the whole guide: architect the AI feature in Mercado. You'll run the complete system —cache, cascade, guardrails, eval, fallback, circuit breaker, shell, and feedback, all together— over varied tickets and a model outage, with the feature's diagram, a complete ADR, and the argument for why the non-determinism stays contained. The sheet's ten fields, executed as a single system.

Resources

• Anthropic, "Building Effective Agents" (2024) — anthropic.com/engineering/building-effective-agents. The distinction between a model that suggests an action and a system that decides to execute it is the axis of safe-agent design; this lesson's shell is that distinction, measured. In English.
• Chip Huyen, AI Engineering (O'Reilly, 2024). Its discussion of the control of an agent's actions and of the data flywheel as an AI system's improvement engine backs the two parts of this lesson. In English.
• Chip Huyen, Designing Machine Learning Systems (O'Reilly, 2022). The chapters on feedback loops and monitoring distinguish the operational metrics (of the service) from the quality ones (of the model) —exactly the distinction of Part B's observability—. In English.
• architecture-for-ai-native-systems-guide, Modules 6 and 7 (this ecosystem) — the in-depth treatment of the deterministic shell (propose/dispose, structured action, capabilities) and of the data loop (flywheel, observability, feedback capture). This lesson integrates into the support agent what M6 and M7 developed. In Spanish.