Bounded capabilities

Overview

Lesson 3 showed that the model proposes a command from a menu, and that the actions outside the menu are rejected by definition —delete_database didn't exist for the shell because it wasn't in ACTION_MENU—. That lesson treated it as a consequence of the format. This lesson turns it into a design principle with its own name: bounded capabilities, the AI-native version of classic security's least privilege principle. The idea is simple and powerful: the model can only propose from a closed and minimal menu of allowed actions, and that menu is designed by giving it as little as possible —only what it needs for its job, and nothing more—.

What makes this lesson different from the previous one is the question it answers: not how what's outside the menu is rejected (that's L3's channel check), but how big the menu should be. And the answer is measured: an agent's blast radius —how many irreversible and dangerous operations a hallucination can reach— is exactly the number of those operations that fall within its menu. With a bounded menu, that number is zero. With a broad menu, "so the agent is useful", each dangerous operation you add is one a hallucination can now reach. You'll see, executed, how the same batch of model proposals —which includes hallucinated destructive actions— leaves 0 dangerous operations within reach with a small menu and 3 with a broad menu.

Connection with the module. This lesson defines what the model can propose, between the structured action (L3, the format) and the rule validation (L5, the policy). It's a gate distinct from the rules: capabilities say "this action doesn't exist for this agent" (a $5000 refund and a delete_account fail for different reasons —the first by policy, the second by capability—). The boundary with the domain guides: what capabilities each agent needs is a system design decision; here we treat the principle (give the minimum) and how to measure its effect. The boundary with the security guide: least privilege in depth —permission models, roles, escalation— is a security topic; here we apply it to an LLM's action menu.

An analogy: the keys you give the new employee

When you hire a new employee at a store, you decide which keys to give them. You could give them the manager's full keyring —the safe, the warehouse, the payroll system, the back door, everything— "so they can help with anything". Or you could give them only the keys their job needs: the cash register and the product storeroom door, say. The two options have an equally capable and well-intentioned employee. The difference is in what they could end up doing when they make a mistake —or when someone tricks them—.

With the full keyring, a confused new employee can, without malice, open the safe and leave it open, move something in the payroll, or give warehouse access to the wrong person. The blast radius of their error is enormous, because they have keys for everything. With the minimal keys, that same employee, equally confused, can't touch the safe or the payroll —they don't have the key—; their error, in the worst case, is a problem at the cash register, bounded and recoverable. You didn't give them fewer keys because you distrust them; you gave them fewer because their error (inevitable, human) does less damage when their access is minimal.

The security principle has a name: least privilege, giving each person the minimum access their function needs. And it applies identically to an LLM. The agent's "keyring" is its capabilities menu. Giving it broad capabilities "so it's useful" is giving it the manager's keyring: each irreversible action you add is a key a hallucination can use. Giving it a minimal menu is giving it just the keys: when the model makes a mistake —and it will— it won't have the safe's key, because you never gave it. Mercado's support agent needs to refund, escalate, and send messages; it doesn't need to change prices, ban sellers, or deactivate accounts. Those keys simply don't go on its keyring.

Worked example: the blast radius grows with the menu

Let's measure the blast radius. We define the capabilities granted to the support agent —a bounded menu: refund, escalate_to_human, send_message— and a set of irreversible and dangerous operations that exist in the platform but that this agent should never touch: delete_account, change_price, ban_user. Then we run a batch of actions the model proposed, which mixes actions within its capabilities with dangerous operations it "hallucinated" being able to do. We measure how many dangerous ones stay within reach with the bounded menu, and compare against a broad menu that gave it all of them "so the agent is useful".

# Module 6, Lesson 4: bounded capabilities (least privilege).
# The support agent can only PROPOSE from a MENU of allowed actions.
# Everything that falls outside the menu is rejected, however much the LLM proposes it.
# No network, no API, no keys. Fixed data.

# Capabilities GRANTED to the support agent: a small, bounded menu.
GRANTED = {"refund", "escalate_to_human", "send_message"}

# IRREVERSIBLE operations that exist in the platform but that this agent
# should NEVER be able to touch.
DANGEROUS = {"delete_account", "change_price", "ban_user"}


def capability_check(action, granted):
    return action in granted


# Batch the probabilistic core (SIMULATED) proposed. It mixes actions within
# its capabilities with dangerous operations it "hallucinated" being able to do.
PROPOSED = [
    "refund",
    "delete_account",
    "change_price",
    "escalate_to_human",
    "ban_user",
    "send_message",
]

print("=== NARROW menu: the agent only has 3 capabilities ===")
print(f"    GRANTED = {sorted(GRANTED)}")
print()
print(f"{'proposed action':<22}{'verdict'}")
print("-" * 42)
within = outside = 0
reached_dangerous_narrow = []
for action in PROPOSED:
    ok = capability_check(action, GRANTED)
    if ok:
        within += 1
        verdict = "ALLOWED (in capabilities)"
    else:
        outside += 1
        verdict = "REJECTED (outside the menu)"
        if action in DANGEROUS:
            pass  # with the narrow menu, no dangerous one was within reach
    print(f"{action:<22}{verdict}")

print()
print(f"  within capabilities : {within}/{len(PROPOSED)}")
print(f"  outside the menu    : {outside}/{len(PROPOSED)}")
print(f"  dangerous operations reachable : {len(reached_dangerous_narrow)}")

# --- Contrast: a BROAD menu "so the agent is useful". ---
GRANTED_BROAD = GRANTED | DANGEROUS
reached_dangerous_broad = [a for a in PROPOSED
                           if a in DANGEROUS and capability_check(a, GRANTED_BROAD)]

print()
print("=== BROAD menu: it was given ALL the capabilities ===")
print(f"    GRANTED_BROAD = {sorted(GRANTED_BROAD)}")
print(f"  dangerous operations now reachable : {len(reached_dangerous_broad)}")
print(f"    -> {sorted(reached_dangerous_broad)}")
print()
print("Rule: the blast radius = how many irreversible operations fall within")
print("the menu. Small menu -> small blast radius.")

What to expect. When you run the file, the output is exactly this:

=== NARROW menu: the agent only has 3 capabilities ===
    GRANTED = ['escalate_to_human', 'refund', 'send_message']

proposed action       verdict
------------------------------------------
refund                ALLOWED (in capabilities)
delete_account        REJECTED (outside the menu)
change_price          REJECTED (outside the menu)
escalate_to_human     ALLOWED (in capabilities)
ban_user              REJECTED (outside the menu)
send_message          ALLOWED (in capabilities)

  within capabilities : 3/6
  outside the menu    : 3/6
  dangerous operations reachable : 0

=== BROAD menu: it was given ALL the capabilities ===
    GRANTED_BROAD = ['ban_user', 'change_price', 'delete_account', 'escalate_to_human', 'refund', 'send_message']
  dangerous operations now reachable : 3
    -> ['ban_user', 'change_price', 'delete_account']

Rule: the blast radius = how many irreversible operations fall within
the menu. Small menu -> small blast radius.

Compare the two menus, because the difference between them is the blast radius.

With the bounded menu, the three dangerous operations are rejected by definition. The model proposed delete_account, change_price, and ban_user —three destructive hallucinations—, and all three were rejected with the same reason: "outside the menu". Not because a business rule evaluated them and decided they were wrong, but because they don't exist for this agent: they're not in GRANTED. The final count is the one that matters: dangerous operations reachable: 0. However much the model hallucinated destructive actions, none stayed within reach, because the agent's keyring doesn't have those keys. The three actions within its capabilities (refund, escalate_to_human, send_message) did pass the check —and will proceed to lesson 5's rule validation—, but the dangerous ones stopped here, at the capabilities door.

With the broad menu, the same three hallucinations now do stay within reach. The only change was the keyring: GRANTED_BROAD includes delete_account, change_price, and ban_user "so the agent is useful". And suddenly, the dangerous operations reachable rise from 0 to 3. The model is identical, it proposed the same; what changed is that now those actions exist for the agent, so a hallucination of change_price to $0.01, or of delete_account, is no longer rejected at the door —it has the key—. This is the measured lesson: the blast radius doesn't depend on how good the model is, but on how many dangerous operations fall within its menu. Widening the menu "just in case" doesn't make the agent safer or smarter; it just gives more keys to its errors.

Zero and three is the whole moral. The bounded menu leaves zero dangerous operations within reach; the broad one, three. No business rule, no better prompt, no additional validation produced that difference: only the size of the menu produced it. That's why bounded capabilities are the module's cheapest and strongest defense: they don't require understanding the policy or verifying against a source of truth —they simply require not giving the agent keys it doesn't need—. An operation that's not on the menu can't fail, can't be hallucinated, can't be exploited: for the agent, it doesn't exist.

Going deeper: least privilege applied to what an LLM can propose

The example showed the effect; it's worth understanding the principle and its design consequences.

Capabilities are a gate distinct from the business rules. It's tempting to think "well, the rule validation (L5) is going to catch the bad actions anyway, why bother bounding the menu?". But the two gates answer different questions and protect against different things. The business rules evaluate an action the agent can take —"this refund, which is indeed an agent action, does it meet the policy?"—. The capabilities decide whether the action is even the agent's —"is delete_account something this agent can propose?"—. A $5000 refund and a delete_account are dangerous for different reasons: the first is a legitimate action with an illegitimate value (caught by the rules); the second is an action the agent should never have been able to propose (caught by the capability). If you only had business rules, you'd have to write a rule for each way each dangerous operation could be wrong —an infinite task—. With bounded capabilities, the operations you don't grant simply never get to need rules: they're rejected earlier.

The minimal menu reduces the surface you have to secure. Each capability you grant is an operation for which you have to think of all the ways it could go wrong, write its validation rules, test its edge cases, and monitor its use. A menu of three actions is three operations to secure; a menu of ten is ten. Least privilege doesn't only reduce the blast radius of hallucinations: it reduces the containment work you have to do. Fewer keys, fewer locks to design. That's why "give it less" isn't a paranoid stance: it's the one that makes the system simpler and safer at the same time, which is a rare and valuable combination.

   The blast radius = IRREVERSIBLE operations within the agent's menu

   NARROW menu (least privilege):           BROAD menu ("just in case"):

   GRANTED = {                              GRANTED = {
     refund,            ◄ needed              refund,
     escalate_to_human,   for its             escalate_to_human,
     send_message,        job                 send_message,
   }                                           delete_account,   ◄ a hallucination
                                               change_price,       now has the
   dangerous within reach: 0                   ban_user,          key to these
                                             }
   a hallucination of                        dangerous within reach: 3
   delete_account does NOT
   have the key -> rejected                  the blast radius grows with
                                             each extra key

Granting capabilities is a design decision, not a default. The most common error isn't giving dangerous capabilities on purpose, but giving them by default —connecting the agent to an API that exposes ten operations and not restricting which ones it can use—. The agent inherits everything the API offers, and its menu ends up being "whatever the system can do" instead of "what this agent needs". The AI-native discipline inverts the default: the menu starts empty, and you add a capability only when you can justify that this agent needs it for its job. It's the difference between "I remove the dangerous ones from a long list" (fragile: something always slips past you) and "I add only the necessary to an empty list" (robust: what you didn't justify isn't there).

Bounding the menu doesn't limit the agent in what matters. The natural objection: "but if I give it fewer capabilities, the agent solves fewer cases". In practice, almost never. The support agent solves 99% of its cases with refunding, escalating, and sending messages; the rare cases that would need change_price or delete_account are precisely the ones that should go through a human —via escalate_to_human—, not through a model hallucination. Bounding the menu doesn't take away the agent's job; it takes away the power to cause damage that isn't even part of its job. And for what it does need and can't do, the right capability is escalate_to_human: the emergency exit that turns "I don't have the key" into "I hand it to whoever has it".

Common mistakes

Giving the agent broad capabilities "so it's useful". What happens: so the agent "can solve anything", it's given a huge menu —refund, give credit, change prices, cancel, deactivate accounts—. Each extra irreversible operation is a key a hallucination can use; the day the model proposes change_price to $0.01 or delete_account, that action was on the menu, so the capability lets it through. Why it happens: "useful" is confused with "powerful", and it's feared that a small menu will limit the agent. How to detect it: count the irreversible operations in your agent's menu; that number is your blast radius. How to fix it: apply least privilege —start with an empty menu and add only what the agent needs for its job, and use escalate_to_human for what it doesn't—. The example measures it: 3 capabilities leave 0 dangerous operations within reach; the broad menu leaves 3.

Inheriting capabilities by default from the API you connect the agent to. What happens: you connect the agent to a service that exposes many operations and don't restrict which ones it can invoke, so it inherits the whole catalog. Its menu ends up being "whatever the system can do". Why it happens: it's the default —not restricting is easier than restricting— and the problem isn't seen until the model hallucinates an operation you never thought it could reach. How to detect it: if you can't enumerate your agent's exact menu in a short and explicit list, it probably inherited too much. How to fix it: define an explicit and minimal GRANTED, and reject everything that isn't in it, regardless of what the underlying API exposes. You define the agent's menu, not the API.

Trusting the prompt to bound what the model can do. What happens: instead of a closed menu in code, the model is told in the prompt "you can only refund, escalate, and send messages; never change prices or deactivate accounts". Most of the time it obeys; but the prompt is a suggestion to a probabilistic component, and the day the model proposes delete_account anyway —because a user manipulated it, or simply from non-determinism—, if the capability exists in the system, it's executed. Why it happens: putting the limit in the prompt is easy and seems to work. How to detect it: if your only barrier against a dangerous action is an instruction in the prompt, you have a probability, not a guarantee. How to fix it: the menu lives in code —if action not in GRANTED: reject—, which holds 100% of the time. The prompt can ask the model to stay within its menu (it helps it propose well), but the guarantee is given by the capability check, not the prompt. It's the same principle as in L1: the policy goes in code, not in the prompt.

Exercises

Exercise 1 — Which keys to give it. You're designing the capabilities menu for three Mercado agents. For each one, propose a minimal menu and say which dangerous operation you would not give it and why: (a) the customer support agent; (b) the "describe your product" generator for sellers; (c) the semantic search assistant.

See solution

• (a) Support agent: minimal menu {refund, escalate_to_human, send_message}. You wouldn't give it change_price or delete_account or ban_user: they aren't part of solving a support case, and if a case ever needed them, it must go to a human via escalate_to_human. Blast radius bounded to refunds (which L5's rules also validate).
• (b) "Describe your product" generator: minimal menu {propose_description} (or {save_draft}). You wouldn't give it publish directly nor set_price: the generator proposes a text; publishing it or setting the price are decisions that go through content validation (M4) and through rules/human, not through the model. Even less would it touch accounts or orders, which have nothing to do with its job.
• (c) Semantic search assistant: minimal menu {search} —or even no action capability at all, because its job is to read and return results, not to do anything—. You wouldn't give it any operation that touches state (no refund, change_price, automatic add_to_cart): a search shouldn't be able to modify anything. Its ideal blast radius is zero state operations.

The pattern: the menu is derived from the agent's job, not from "what could end up being useful". What the agent doesn't need for its job doesn't go on its keyring; and for the rare thing it would need, the right capability is usually to escalate to a human.

Exercise 2 — The blast radius. In the example, the bounded menu left 0 dangerous operations within reach and the broad one left 3. Suppose the model improves and hallucinates destructive actions half as often as before. How does each menu's blast radius change? Use the answer to argue why bounding the menu is a better defense than improving the model.

See solution

The blast radius doesn't change with the model's quality: it's still 0 for the bounded menu and 3 for the broad one. The blast radius is how many dangerous operations fall within the menu, and that depends on the menu, not on how often the model hallucinates. Improving the model reduces how many times it proposes a destructive action, but doesn't change which ones it can reach: if delete_account is on the menu, a single hallucination —however rare— reaches it; if it's not, none reaches it, however often the model hallucinates.

The argument: improving the model lowers the frequency of the error, but at scale any positive frequency ends up occurring, and a single execution of delete_account is a disaster. Bounding the menu, on the other hand, brings the blast radius to zero hard, independent of the frequency: the dangerous operation doesn't exist for the agent. That's why bounded capabilities are a categorical defense (eliminates the possibility) and "better model" is a statistical defense (reduces the probability). Near irreversible operations, you want the categorical one. It's the same argument as module 1's shell: the guarantee is given by the containment, not by the core's quality.

Exercise 3 — Capability or rule. For each of these cases, say whether the action should be blocked by capability (it's not on the agent's menu) or by business rule (it's on the menu but violates the policy), and explain the difference: (a) the support agent proposes delete_account; (b) the support agent proposes a refund of $5000; (c) the support agent proposes a refund of an already-refunded order; (d) the support agent proposes change_price.

See solution

• (a) delete_account → capability. Deactivating accounts isn't part of the support agent's job; it's not on its menu. It's rejected because the action doesn't exist for this agent, not because a rule evaluates it. Capabilities gate (this lesson).
• (b) refund of $5000 → business rule. Refunding is an agent action (it's on the menu); what's wrong is the value —$5000 exceeds the limit—. The action is legitimate; the amount isn't. Business-rules gate (lesson 5).
• (c) refund of an already-refunded order → business rule. Again, refund is on the menu; what violates the policy is the state of the order (already refunded). The action is the agent's; the circumstances block it. Rules gate (lesson 5).
• (d) change_price → capability. Changing prices isn't part of the support agent's job; it's not on its menu. It's rejected as a capability, just like (a).

The key difference: capabilities ask "is this action the agent's?" (L4 gate); the rules ask "does this agent action meet the policy?" (L5 gate). (a) and (d) fail because the action isn't the agent's; (b) and (c) fail because, being the agent's, they violate the policy. A good design has both gates: the capabilities trim what it can propose, the rules validate how.

Summary and next step

In this lesson you turned lesson 3's menu into a design principle: bounded capabilities, least privilege applied to what an LLM can propose. The model can only propose from a closed and minimal menu; what isn't on it is rejected by definition. You measured it: the same batch of model proposals —with three destructive hallucinations— left 0 dangerous operations within reach with a bounded menu and 3 with a broad menu "so the agent is useful". You saw that the blast radius depends on the size of the menu, not on the model's quality; that capabilities are a gate distinct from the business rules (an action not on the menu vs an action on the menu violating the policy); that the menu is designed starting empty and adding only the necessary; and that bounding doesn't limit the agent in what matters, because the rare thing it would need goes through escalate_to_human.

Before moving on you should be able to: state the least privilege principle and apply it to an agent's menu; explain why the blast radius is the number of dangerous operations within the menu; distinguish a block by capability from one by business rule; and argue why the menu goes in code and not in the prompt.

Lesson 5 reaches the heart of the module: validating the proposal against the business rules. You already know that the model proposes a well-formed command (L3) from a bounded menu (L4); now, for the actions that are the agent's —a refund—, you have to decide whether they meet the policy before executing. You'll see, executed with a rule-by-rule matrix, how each refund proposal is validated against the policy's five rules —order exists, not refunded, within the window, amount ≤ total, amount ≤ limit— and is executed only if it passes them all. It's the gate that caught the $5000 refund from the beginning of the guide, now opened rule by rule.

Resources

• The least privilege principle, a pillar of classic security (Saltzer and Schroeder, 1975): give each component the minimum set of permissions it needs for its function, and nothing more. This lesson applies it to an LLM's action menu. Any introductory security reference or the OWASP material covers it. In English.
• Anthropic, "Building Effective Agents" (2024) — anthropic.com/engineering/building-effective-agents. Its recommendation to give the agent a bounded and well-defined set of tools, instead of broad access, is exactly this lesson's bounded-capabilities principle. In English.
• Claude documentation, tool use — docs.anthropic.com. By defining the tools a model can invoke, you define its capabilities menu; the practice of exposing only the necessary tools is least privilege in action. Without focusing on a specific model version. In English.
• Martin Fowler and Bharani Subramaniam, "Emerging Patterns in Building GenAI Apps" — martinfowler.com/articles/gen-ai-patterns. The treatment of an agent's limits and permissions places bounded capabilities within the map of containment patterns. In English.