Measuring a pipeline's coordination cost

Description

Lesson 04 ran Reservo's complete pipeline and left a loose observation: at no point across the three stages did anyone need to decide "which one's next?". This lesson turns that observation into a number. You'll take the SAME three-stage task — quote, validate the cancellation policy, confirm — and calculate how much it would cost to solve if, instead of a pipeline with a fixed order, a Module-2-style supervisor had to decide, from scratch, whose turn it is after every result — exactly what a supervisor would do if it had no "fixed order" concept built in at all.

The result isn't an opinion: it's a count. You'll see this module's pipeline saves exactly one routing call per stage — three fewer model calls, for this three-step task — and four fewer hops, without losing a single tool call or changing the final answer. This is the measurement holding up lesson 01's line: "fewer coordination calls, fewer decision points that can go wrong."

Connection to the module

This lesson reuses, unchanged, the pipeline run in lesson 04 — same PIPELINE_STAGES, same scripts, same run_pipeline. The only new thing is the cost model comparing that real result against a hypothetical supervisor solving the same sequence. Lesson 07 picks up this same comparison from another angle: not how much the fixed order costs, but whether that fixed order was, to begin with, the right choice for every stage.

Analogy: asking for directions at every corner, versus following an already-drawn map

Picture arriving in a new city with two ways to move between three places you know you're going to visit, always in the same order. The first: at every corner, asking someone "which way now?" — even though the final destination is always the same, every question costs time, and every answer is one more chance of being pointed the wrong way. The second: following a map you already drew before leaving, with the complete route marked — you never ask anything, because you already knew, from the start, you were going to pass through those three places in that order. This module's pipeline is the already-drawn map; a supervisor solving the same sequence, step by step, is asking at every corner.

Worked example: the same pipeline, with the cost counted

import ast
import concurrent.futures
from dataclasses import dataclass
import reservo_tools as rt


def dispatch_parallel(tool_use_blocks, tools):
    with concurrent.futures.ThreadPoolExecutor(max_workers=len(tool_use_blocks)) as pool:
        futures = [pool.submit(tools[b["name"]], **b["input"]) for b in tool_use_blocks]
        results = [f.result() for f in futures]
    return [
        {"type": "tool_result", "tool_use_id": b["id"], "content": str(r)}
        for b, r in zip(tool_use_blocks, results)
    ]


def run_agent_parallel(question, model_script, tools, max_iterations=10):
    messages = [{"role": "user", "content": question}]
    for step in range(max_iterations):
        turn = model_script[step]
        messages.append({"role": "assistant", "content": turn["content"]})
        if turn["stop_reason"] != "tool_use":
            return turn, messages
        tool_result_blocks = dispatch_parallel(turn["content"], tools)
        messages.append({"role": "user", "content": tool_result_blocks})
    raise RuntimeError(f"max_iterations reached ({max_iterations})")


POLICY_DOCS = {
    "cancellation-policy": (
        "Bookings can be cancelled at no charge up to 2 hours before the "
        "booked time slot. Cancellations within those 2 hours incur the "
        "no-show fee."
    ),
}


def search_docs(query):
    q = query.lower()
    if "cancel" in q:
        return f"[cancellation-policy] {POLICY_DOCS['cancellation-policy']}"
    return "No relevant policy was found for that question."


SPECIALISTS = {
    "booking_agent": {"tools": {"get_quote": rt.get_quote, "book_room": rt.book_room}},
    "policy_agent": {"tools": {"search_docs": search_docs}},
}


def run_specialist(name, task, model_script):
    tools = SPECIALISTS[name]["tools"]
    return run_agent_parallel(task, model_script, tools)


@dataclass
class PipelineStage:
    kind: str
    name: str
    label: str


def last_tool_result(history):
    for m in reversed(history):
        if isinstance(m["content"], list):
            for b in m["content"]:
                if b["type"] == "tool_result":
                    try:
                        return ast.literal_eval(b["content"])
                    except (ValueError, SyntaxError):
                        return b["content"]
    return None


def build_stage_task(stage, payload):
    if stage.kind == "quote":
        return f"Quote {payload['room']} {payload['tier']} {payload['hours']}h."
    if stage.kind == "validate_policy":
        return (f"What's the cancellation policy for a {payload['hours']}h "
                 f"{payload['room']} booking, before confirming it?")
    if stage.kind == "confirm":
        return (f"Book {payload['room']} {payload['tier']} {payload['hours']}h "
                 f"for {payload['member']} -- the cancellation policy has "
                 f"already been validated.")
    raise ValueError(f"don't know how to build the task for stage {stage.kind!r}")


def extract_payload(stage, history, payload):
    new_payload = dict(payload)
    result = last_tool_result(history)
    if stage.kind == "quote":
        new_payload["price_cents"] = result["price_cents"]
    elif stage.kind == "validate_policy":
        new_payload["cleared_to_book"] = True
    elif stage.kind == "confirm":
        new_payload["booking_id"] = result["booking_id"]
        new_payload["confirmed"] = result["confirmed"]
    return new_payload


def run_pipeline(stages, model_scripts, initial_payload):
    payload = dict(initial_payload)
    trace = []
    for i, stage in enumerate(stages):
        task = build_stage_task(stage, payload)
        final, history = run_specialist(stage.name, task, model_scripts[i])
        payload = extract_payload(stage, history, payload)
        trace.append({"stage": i + 1, "label": stage.label, "history": history})
    return payload, trace


PIPELINE_STAGES = [
    PipelineStage(kind="quote", name="booking_agent", label="quote"),
    PipelineStage(kind="validate_policy", name="policy_agent",
                  label="validate the cancellation policy"),
    PipelineStage(kind="confirm", name="booking_agent", label="confirm the booking"),
]
INITIAL_PAYLOAD = {"room": "Focus", "tier": "pro", "hours": 3, "member": "Ana"}

model_script_quote = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "get_quote",
         "input": {"room": "Focus", "tier": "pro", "hours": 3}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "Focus pro 3h costs 6000 cents."}]},
]
model_script_policy = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "search_docs",
         "input": {"query": "cancellation policy"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": ("You can cancel at no charge up to 2 hours before the "
                                    "booked time. There's no impediment to confirming.")}]},
]
model_script_confirm = [
    {"stop_reason": "tool_use", "content": [
        {"type": "tool_use", "id": "toolu_01", "name": "book_room",
         "input": {"room": "Focus", "tier": "pro", "hours": 3, "member": "Ana"}}]},
    {"stop_reason": "end_turn", "content": [
        {"type": "text", "text": "I booked Focus pro 3h for Ana (confirmation #1)."}]},
]

model_scripts = [model_script_quote, model_script_policy, model_script_confirm]
payload, trace = run_pipeline(PIPELINE_STAGES, model_scripts, INITIAL_PAYLOAD)

specialist_calls = sum(1 for s in trace for m in s["history"] if m["role"] == "assistant")
specialist_tools = sum(
    1 for s in trace for m in s["history"] if isinstance(m["content"], list)
    for b in m["content"] if b["type"] == "tool_use"
)
N = len(PIPELINE_STAGES)
COMPOSE_CALLS = 1  # concept: the final synthesis for the member (lesson 04)

# --- Pipeline (this module): the order is fixed, zero routing calls ---
pipeline_route_calls = 0
pipeline_total = pipeline_route_calls + specialist_calls + COMPOSE_CALLS
pipeline_hops = N - 1  # DIRECT handoff from one stage to the next

# --- Hypothetical: a Module-2-style supervisor solving the SAME 3-stage
# sequence, deciding from scratch whose turn it is after every result --
# because, unlike the pipeline, it has no order built in beforehand ---
supervisor_route_calls = N       # one routing decision before EACH stage
supervisor_total = supervisor_route_calls + specialist_calls + COMPOSE_CALLS
supervisor_hops = 2 * N          # round trip to each specialist (M2 L06 convention)

print(f"{'':36}{'Pipeline (M3)':>16}{'Repeated supervisor':>22}")
print(f"{'model calls (routing)':36}{pipeline_route_calls:>16}{supervisor_route_calls:>22}")
print(f"{'model calls (specialists)':36}{specialist_calls:>16}{specialist_calls:>22}")
print(f"{'model calls (synthesis)':36}{COMPOSE_CALLS:>16}{COMPOSE_CALLS:>22}")
print(f"{'model calls TOTAL':36}{pipeline_total:>16}{supervisor_total:>22}")
print(f"{'hops between agents':36}{pipeline_hops:>16}{supervisor_hops:>22}")

extra_calls = supervisor_total - pipeline_total
extra_hops = supervisor_hops - pipeline_hops
print()
print(f"difference: the repeated supervisor uses {extra_calls} MORE model calls "
      f"({extra_calls}/{N} = exactly 1 per stage) and {extra_hops} MORE hops than "
      f"the pipeline -- for the SAME {N}-stage task, with the SAME {specialist_calls} "
      f"internal calls and the SAME {specialist_tools} tool calls.")

What to expect:

                                       Pipeline (M3)  Repeated supervisor
model calls (routing)                             0                    3
model calls (specialists)                         6                    6
model calls (synthesis)                           1                    1
model calls TOTAL                                 7                   10
hops between agents                               2                    6

difference: the repeated supervisor uses 3 MORE model calls (3/3 = exactly 1 per stage) and 4 MORE hops than the pipeline -- for the SAME 3-stage task, with the SAME 6 internal calls and the SAME 3 tool calls.

The pipeline solves the same task with 7 model calls; a supervisor that had to decide at each of the three stages would spend 10 — 30% more, purely in coordination, without either path doing more real work: the same 6 internal specialist calls, the same 3 tool calls, the same final information available.

Where each number comes from

It's worth breaking down the count, because every piece has a concrete reason:

Pipeline:
  0 routing calls   -- PIPELINE_STAGES already carries the fixed order; run_pipeline
                        never evaluates "which one's next?"
  6 internal calls  -- 2 per stage (one tool_use, one final text) x 3 stages
  1 synthesis       -- concept, the final answer for the member (lesson 04)
  ------------------------------------------------------------------
  7 TOTAL

Repeated supervisor (hypothetical, NOT built in this module):
  3 routing calls   -- one decision before EACH stage (N = 3), because a
                        supervisor with no fixed order has to decide again
                        "whose turn is it now?" after every result
  6 internal calls  -- IDENTICAL to the pipeline -- the real work doesn't change
  1 synthesis       -- IDENTICAL to the pipeline
  ------------------------------------------------------------------
  10 TOTAL

Hops follow the same logic. In the pipeline, every stage hands its payload directly to the next one — N - 1 = 2 handoffs for 3 stages, with none of them going back through a central coordinator. In the repeated supervisor, every specialist is consulted with a round trip — the same convention Module 2, lesson 06 already used: 2 hops per specialist consulted — so 3 specialists consulted independently cost 2 * 3 = 6 hops.

Why the saving is "exactly 1 per stage"

It isn't a coincidence of this specific task — it's a general property of the pattern. A supervisor that doesn't know the order is fixed has to spend one decision (concept) on every transition, no matter how many stages the sequence has: with 3 stages, 3 decisions; with 5, it would be 5. A pipeline, by contrast, always spends zero decisions, no matter how long the sequence is, because the complete order was already written in PIPELINE_STAGES before the first request. The saving grows at exactly the same rate the pipeline grows — a concrete, measurable reason to prefer this pattern when the sequence of steps genuinely never changes.

Common mistakes

1. Thinking this module actually built a "repeated supervisor." It didn't — it's a cost model, not a running system. Module 2 never built a supervisor that re-decided at every stage of a fixed sequence; this lesson only calculates how much it would cost if someone did, so it can be compared against lesson 04's real pipeline.

2. Adding up the total cost wrong. pipeline_route_calls + specialist_calls + COMPOSE_CALLS isn't the same as specialist_tools — mixing up "model calls" with "tool calls" leads to wrong comparisons, the same mistake already flagged in Module 2, lesson 06.

3. Generalizing this task's 30% to any pipeline. The exact percentage depends on how many internal calls each stage has — here, 2 per stage. What does generalize is the pattern: the saving in routing calls is always N (one per stage), no matter how much internal work each stage has.

4. Concluding a pipeline is always "better" than a supervisor. This comparison measures the cost of coordinating the SAME fixed sequence two different ways — it doesn't compare pipeline against supervisor in general. When the order of the steps CAN genuinely vary depending on the request (Module 2's central case), a pipeline doesn't apply at all: there's no "fixed order" to hardcode.

5. Forgetting the pipeline's hops are direct handoffs, not round trips. The N - 1 formula assumes every stage hands its payload directly to the next one, without going back through any central coordinator in between — exactly how run_pipeline has been built since lesson 03.

Exercises

Exercise 1: Recalculate the saving for a 5-stage pipeline (Easy)

Without running any code yet, use this lesson's pattern to predict: if a pipeline had 5 stages, each with 2 internal calls, how many routing calls would it save over a repeated supervisor? How many hops would it save?

def pipeline_savings(n_stages, calls_per_stage=2):
    pipeline_calls = 0 + (calls_per_stage * n_stages) + 1
    supervisor_calls = n_stages + (calls_per_stage * n_stages) + 1
    pipeline_hops = n_stages - 1
    supervisor_hops = 2 * n_stages
    return {
        "calls saved": supervisor_calls - pipeline_calls,
        "hops saved": supervisor_hops - pipeline_hops,
    }

print(pipeline_savings(5))

{'calls saved': 5, 'hops saved': 6}

Exercise 2: Measure the cost of a single-stage pipeline (Medium)

Using Exercise 1's formula, calculate the saving for a pipeline with just one stage. Does the result make sense? Relate it to what you already know from Module 1, lesson 05.

print(pipeline_savings(1))

{'calls saved': 1, 'hops saved': 2}

Exercise 3: When does the saving stop mattering? (Hard)

Reservo processes 500 bookings a day, each one going through this 3-stage pipeline. Calculate how many model calls the pipeline saves over the repeated supervisor, per day. Then, in prose, argue: in what scenario would that daily saving stop being the decisive criterion for choosing one pattern over the other?

DAILY_BOOKINGS = 500
savings_per_booking = pipeline_savings(3)["calls saved"]
print(f"daily saving: {DAILY_BOOKINGS * savings_per_booking} model calls")

daily saving: 1500 model calls

Summary and next step

• We measured, with real numbers, the cost of coordinating the same 3-stage pipeline two ways: with this module's fixed order (7 model calls, 2 hops) and with a hypothetical supervisor that decided at every stage (10 calls, 6 hops).
• The saving is exactly 1 routing call per stage — a general property of the pattern, not a coincidence of this specific task — and 4 fewer hops, without changing either the specialists' internal work or the final answer.
• The saving scales linearly with the number of stages — more stages in the sequence, more calls saved, as long as the order is genuinely fixed.
• This saving never justifies forcing a pipeline onto a task that actually needs to decide — the same "measure before generalizing" principle from this whole guide.

Next lesson: 06 — When a stage fails. With the cost already measured, we look at the other side of having zero decisions in the middle: what happens when a stage has no good result to hand off to the next one.

Additional resources

1. Anthropic — Multi-agent research system — Anthropic's report on the real cost of coordinating, the same kind of measurement this lesson runs by hand over Reservo's pipeline.
2. Anthropic — Building effective agents — The principle of using the simplest mechanism that solves the task — a fixed-order pipeline is, exactly, that simplest mechanism when the sequence never changes.
3. Python — Functions and default argument values — The basis for pipeline_savings, this lesson's function that generalizes the count to any number of stages.
4. Python 3.14 — What's New — The version every line of this measurement ran on.