Measuring the latency savings in rounds

Description

Lessons 03 through 06 left two complete paths for the same compound request: run_fanout_sequential, which dispatches one sub-task after another, and run_fanout_parallel, which dispatches them all at once with ThreadPoolExecutor. This lesson answers the question that justifies lesson 06's existence: how much does running in parallel actually save?

The answer has a part that's surprising if you're coming from Module 3: the fan-out doesn't save a single model call. The same calls — the split, each specialist's internal calls, the synthesis — happen on both paths, exactly the same ones. What changes is how many rounds it takes to complete them: in sequential, the rounds get added up; in parallel, they end up bounded by the sub-task that needs the most, because the rest advance at the same time. This lesson measures that difference with a round count — never with time.time() or any real clock — and builds a formula that generalizes the result to any number of sub-tasks.

Connection to the module

This lesson reuses the numbers already produced in lesson 04 (each specialist's internal calls) and lesson 06's mechanism (run_fanout_parallel), without modifying either one. What's new is the cost model — fanout_rounds — that turns those numbers into a rounds comparison. Lesson 08 (mini-project) applies this same formula to new scenarios, including one with three agents.

Analogy: three kitchen tasks, one kitchen or three kitchens

Picture preparing three dishes that don't depend on each other — a salad, a soup, a dessert — each with its own steps. With a single kitchen (sequential), you prepare the complete salad, then the complete soup, then the complete dessert: the total time is the sum of the three. With three kitchens (parallel), you put someone in each one at the same time: the total time is no longer the sum — it's the time of the slowest dish of the three, because the other two would already be done by the time that one finishes. If the salad takes 2 steps, the soup 2 steps, and the dessert 2 steps, with one kitchen that's 6 steps total; with three kitchens, it's 2 steps — bounded by the slowest one, which in this case is all three equally.

Worked example: sequential vs. parallel rounds, with real numbers

import concurrent.futures
from dataclasses import dataclass
import reservo_tools as rt


def dispatch_parallel(tool_use_blocks, tools):
    with concurrent.futures.ThreadPoolExecutor(max_workers=len(tool_use_blocks)) as pool:
        futures = [pool.submit(tools[b["name"]], **b["input"]) for b in tool_use_blocks]
        results = [f.result() for f in futures]
    return [
        {"type": "tool_result", "tool_use_id": b["id"], "content": str(r)}
        for b, r in zip(tool_use_blocks, results)
    ]


def run_agent_parallel(question, model_script, tools, max_iterations=10):
    messages = [{"role": "user", "content": question}]
    for step in range(max_iterations):
        turn = model_script[step]
        messages.append({"role": "assistant", "content": turn["content"]})
        if turn["stop_reason"] != "tool_use":
            return turn, messages
        tool_result_blocks = dispatch_parallel(turn["content"], tools)
        messages.append({"role": "user", "content": tool_result_blocks})
    raise RuntimeError(f"max_iterations reached ({max_iterations})")


POLICY_DOCS = {
    "cancellation-policy": (
        "Bookings can be cancelled at no charge up to 2 hours before the "
        "booked time slot. Cancellations within those 2 hours incur the "
        "no-show fee."
    ),
}


def search_docs(query):
    q = query.lower()
    if "cancel" in q:
        return f"[cancellation-policy] {POLICY_DOCS['cancellation-policy']}"
    return "No relevant policy was found for that question."


SPECIALISTS = {
    "booking_agent": {
        "tools": {
            "list_rooms": rt.list_rooms, "get_quote": rt.get_quote,
            "book_room": rt.book_room, "cancel_booking": rt.cancel_booking,
        },
    },
    "policy_agent": {"tools": {"search_docs": search_docs}},
}


def run_specialist(name, task, model_script):
    tools = SPECIALISTS[name]["tools"]
    return run_agent_parallel(task, model_script, tools)


def count_model_calls(history):
    return sum(1 for m in history if m["role"] == "assistant")


@dataclass
class SubTask:
    agent: str
    task: str


def run_fanout_sequential(subtasks, model_scripts):
    ordered = sorted(subtasks, key=lambda s: s.agent)
    results = {}
    for sub in ordered:
        final, history = run_specialist(sub.agent, sub.task, model_scripts[sub.agent])
        results[sub.agent] = {"task": sub.task, "output": final["content"][0]["text"], "history": history}
    return results


subtasks = [
    SubTask(agent="booking_agent", task="Quote Focus pro 3h."),
    SubTask(agent="policy_agent", task="What's the cancellation policy?"),
]
model_scripts = {
    "booking_agent": [
        {"stop_reason": "tool_use", "content": [
            {"type": "tool_use", "id": "toolu_01", "name": "get_quote",
             "input": {"room": "Focus", "tier": "pro", "hours": 3}}]},
        {"stop_reason": "end_turn", "content": [
            {"type": "text", "text": "Focus pro 3h costs 6000 cents."}]},
    ],
    "policy_agent": [
        {"stop_reason": "tool_use", "content": [
            {"type": "tool_use", "id": "toolu_01", "name": "search_docs",
             "input": {"query": "cancellation policy"}}]},
        {"stop_reason": "end_turn", "content": [
            {"type": "text", "text": (
                "You can cancel at no charge up to 2 hours before the "
                "booked time. After that, the no-show fee applies."
            )}]},
    ],
}

results = run_fanout_sequential(subtasks, model_scripts)
call_counts = {a: count_model_calls(results[a]["history"]) for a in results}
print("--- internal calls per specialist (identical on both paths) ---")
for a in sorted(call_counts):
    print(f"  {a:15} -> {call_counts[a]} model calls")

FANOUT_SPLIT_CALLS = 1   # concept: the supervisor recognizes the sub-tasks -- happens ONCE, before dispatching
COMPOSE_CALLS = 1        # concept: the final synthesis -- happens ONCE, AFTER both finish
HOPS_PER_SPECIALIST = 2  # M2 L06 / M3 L05 convention: a round trip to each specialist


def fanout_rounds(call_counts_list, surrounding_calls):
    """A 'round' is one round-trip step with the model -- NEVER real
    wall-clock time (there's no time.time() in this guide). Sequential:
    each specialist waits for the previous one to finish, so the rounds
    get ADDED UP. Parallel: the specialists run at the same time, so the
    rounds end up bounded by the one that needs the MOST -- max(), not
    sum()."""
    sequential = sum(call_counts_list) + surrounding_calls
    parallel = max(call_counts_list) + surrounding_calls
    return sequential, parallel


counts_list = list(call_counts.values())
seq_rounds, par_rounds = fanout_rounds(counts_list, FANOUT_SPLIT_CALLS + COMPOSE_CALLS)
total_calls = sum(counts_list) + FANOUT_SPLIT_CALLS + COMPOSE_CALLS  # the TOTAL number of calls is the same on both paths
total_hops = HOPS_PER_SPECIALIST * len(counts_list)                  # the hops don't change either

print()
print(f"{'':32}{'sequential fan-out':>20}{'parallel fan-out':>20}")
print(f"{'model calls TOTAL':32}{total_calls:>20}{total_calls:>20}")
print(f"{'hops between agents':32}{total_hops:>20}{total_hops:>20}")
print(f"{'coordination rounds':32}{seq_rounds:>20}{par_rounds:>20}")

rounds_saved = seq_rounds - par_rounds
pct_saved = round(100 * rounds_saved / seq_rounds)
print()
print(f"difference: SAME {total_calls} model calls and SAME {total_hops} hops on both "
      f"paths -- the only thing that changes is ROUNDS: {rounds_saved} fewer with real concurrency "
      f"({pct_saved}% fewer), because booking_agent and policy_agent advance their 2 internal "
      f"rounds at the same time, instead of one behind the other.")

What to expect:

--- internal calls per specialist (identical on both paths) ---
  booking_agent   -> 2 model calls
  policy_agent    -> 2 model calls

                                  sequential fan-out    parallel fan-out
model calls TOTAL                                  6                   6
hops between agents                                4                   4
coordination rounds                                6                   4

difference: SAME 6 model calls and SAME 4 hops on both paths -- the only thing that changes is ROUNDS: 2 fewer with real concurrency (33% fewer), because booking_agent and policy_agent advance their 2 internal rounds at the same time, instead of one behind the other.

There's this lesson's central result: 6 = 6 model calls, 4 = 4 hops — neither path saves even a single call or a single hop, because the same decisions have to be made either way. The only column that differs is rounds: 6 in sequential (the split, plus the 2+2 internal calls added up, plus the synthesis), 4 in parallel (the split, plus max(2, 2) = 2 because the two run at the same time, plus the synthesis).

Where each number comes from

Sequential:
  1 split       -- happens once, before anything gets dispatched
  2 + 2 = 4     -- booking_agent and policy_agent, ONE AFTER THE OTHER (added up)
  1 synthesis   -- happens once, after the last one finishes
  ------------------------------------------------------------
  6 ROUNDS

Parallel:
  1 split       -- IDENTICAL -- still happens once, before dispatch
  max(2, 2) = 2 -- booking_agent and policy_agent advance THEIR 2 rounds AT THE SAME TIME
  1 synthesis   -- IDENTICAL -- still waits for BOTH to finish
  ------------------------------------------------------------
  4 ROUNDS

The split and the synthesis don't parallelize with anything — the split has to happen before any sub-task exists to dispatch, and the synthesis has to wait for all the sub-tasks to have finished (lesson 04 already confirmed this with compose_fanout_response's KeyError over incomplete results). That's why surrounding_calls gets added the same way on both paths — the only thing max() instead of sum() changes is the part that does run at the same time: the specialists' internal calls.

Why the fan-out doesn't save calls (unlike the pipeline)

It's worth comparing this to Module 3, lesson 05, which did measure a call saving: there, a pipeline saved exactly one routing call per stage, because the fixed order eliminated a decision a repeated supervisor would otherwise have to pay for. Here there's no decision to eliminate — the split (recognizing there are two independent sub-tasks) has to happen either way, whether you dispatch them sequentially or in parallel. The fan-out doesn't reduce how much decision work is needed; it reduces how much conceptual-clock time (rounds) it takes to complete that same work, because part of it stops waiting in line.

Common mistakes

1. Looking for a model-call saving where there isn't one. This lesson's result isn't "fan-out is cheaper" — it's "fan-out is faster, at the same cost." Confusing the two leads to expecting a number this pattern, by design, doesn't produce.

2. Using sum() for the parallel path, or max() for the sequential one. This is the most direct formula mistake — swapping them produces nonsensical numbers (a parallel path "slower" than sequential, for example). The rule is fixed: sequential adds up, parallel gets bounded by the maximum.

3. Forgetting to add surrounding_calls on both sides equally. The split and the synthesis don't disappear on either path — omitting them from one of the two sides of the comparison overstates the real saving.

4. Generalizing this example's 33% to any fan-out. The percentage depends on how many internal calls each sub-task has and how many sub-tasks there are. What does generalize is the formula (sum() vs. max()) — Exercise 1 applies it to a three-sub-task case with a different result.

5. Thinking the rounds saving is the same as a real wall-clock time saving. It isn't — this guide never measures time.time(). "Rounds" is a discrete count of coordination steps, useful for comparing patterns against each other, not a measurement of a real API's latency.

Exercises

Exercise 1: Fan-out of three agents, each with 2 internal calls (Easy)

Using fanout_rounds, calculate the sequential and parallel rounds for a fan-out of three sub-tasks, each with 2 internal calls (like booking_agent, policy_agent, and pricing_agent in lesson 08's mini-project), with the same worked example's surrounding_calls = 2.

seq_3, par_3 = fanout_rounds([2, 2, 2], surrounding_calls=2)
print(f"sequential rounds: {seq_3}  |  parallel rounds: {par_3}  |  saved: {seq_3 - par_3}")

sequential rounds: 8  |  parallel rounds: 4  |  saved: 4

Exercise 2: Two agents with DIFFERENT internal costs (Medium)

Calculate the sequential and parallel rounds for two sub-tasks with internal costs of 2 and 3 calls respectively (imagine policy_agent needed an extra refinement step in its search), with surrounding_calls = 2.

seq_2, par_2 = fanout_rounds([2, 3], surrounding_calls=2)
print(f"sequential rounds: {seq_2}  |  parallel rounds: {par_2}  |  saved: {seq_2 - par_2}")
print("note: parallel ends up BOUNDED by the slowest specialist (3), not by the average")

sequential rounds: 7  |  parallel rounds: 5  |  saved: 2

Exercise 3: Daily savings over 500 compound requests (Hard)

Reservo receives 500 compound requests per day, each with the same shape as the worked example (two sub-tasks with 2 calls each). Calculate the rounds saved per day if ALL of them were dispatched with run_fanout_parallel instead of run_fanout_sequential.

DAILY_REQUESTS = 500
seq_daily, par_daily = fanout_rounds([2, 2], surrounding_calls=2)
savings_per_request = seq_daily - par_daily
print(f"saving per request: {savings_per_request} rounds")
print(f"daily saving: {DAILY_REQUESTS * savings_per_request} rounds")

saving per request: 2 rounds
daily saving: 1000 rounds

Summary and next step

• The fan-out doesn't save model calls or hops — the same decisions happen on both paths, sequential and parallel, in the exact same quantity.
• What it does save is coordination rounds: sequential adds them up (each sub-task waits for the previous one); parallel bounds them by the maximum (all advance at the same time, limited only by the slowest one).
• For the worked example's request: 6 sequential rounds, 4 parallel rounds — a saving of 2 rounds (33%), with the same 6 model calls and the same 4 hops on both paths.
• The fanout_rounds formula generalizes to any number of sub-tasks and to different internal costs between them — the parallel path always ends up bounded by the slowest sub-task, never by an average.

Next lesson: 08 — Mini-project: fan-out across Reservo. We apply the complete pattern — identifying independence, dispatching (sequential and parallel), aggregating, measuring the saving — to three new scenarios, including one that is NOT fan-out between agents.

Additional resources

1. Anthropic — Building effective agents — The principle that parallelizing independent sub-tasks reduces perceived latency without changing the total work — the exact result this lesson measures.
2. Anthropic — Multi-agent research system — Anthropic's report explicitly measures that a multi-agent system's latency is bounded by the slowest sub-agent when work is split in parallel — the same max() property this lesson confirms.
3. Python — Built-in functions sum() and max() — The two functions that completely distinguish fanout_rounds's sequential cost model from the parallel one.
4. Python 3.14 — What's New — The version every line of this measurement ran on.