Retries with Bounded Backoff

Description

agent-fundamentals Module 7 (Lesson 6) built call_with_retries: a simple retry, with a hard cap, that tells a transient failure apart from a validation error and never retries the second. That same lesson deliberately left one piece unresolved — quoting its closing verbatim: "retrying immediately, with no gap between attempts, can worsen the congestion that caused the failure in the first place [...] this lesson doesn't implement it in detail [...] it's worth keeping in mind for any system that grows beyond a learning exercise". This module is exactly that grown-up system. This lesson closes that pending piece: it builds retry_with_backoff, the direct scale-up of call_with_retries, adding the growing wait between one attempt and the next — calculated, shown, never actually slept.

Connection to the module

This lesson reuses call_with_retries's complete criteria — retry only what's transient, with a hard cap, re-raise the real error if the cap runs out — and adds a single new piece: how long to wait between one attempt and the next. The result, retry_with_backoff, is the function you're going to reuse, unchanged, in this module's Lesson 6 for Claude's 429, and that Lesson 4 is going to wrap with the circuit breaker.

Analogy: ringing the doorbell with growing gaps, not on repeat

Imagine you ring a doorbell and nobody answers. Ringing again a second later, and another second later, and another — is exactly what a retry with no backoff does: insistent, but with no criteria at all for how much time is reasonable to give the person on the other side to reach the door. Someone more patient would wait a little longer after the first failed attempt, a little longer still after the second, a little more each time — giving the other side more time to react on every round, instead of hammering the doorbell. That progression — waiting longer, each time, after every failure — is exactly what exponential backoff models: delay = base * 2**intento. Doubling the wait on every round isn't arbitrary — it's the simplest way to give a saturated service more and more room to recover, instead of piling on more pressure exactly when it needs it least.

Worked example: retry_with_backoff, built and run

Backoff, modeled — never actually slept

# resilience/tool_circuit_breaker.py

def compute_backoff_ms(attempt, base_delay_ms=100):
    """delay = base * 2**attempt -- MODELADO: se calcula y se muestra, nunca
    se duerme de verdad (nunca time.sleep())."""
    return base_delay_ms * (2 ** attempt)


def retry_with_backoff(fn, *args, max_retries=3, base_delay_ms=100,
                        retry_on=(ConnectionError,), **kwargs):
    """Reintenta SOLO las excepciones en retry_on, con un tope duro, y
    calcula (sin dormir) el backoff exponencial de cada intento."""
    last_exc = None
    for attempt in range(1, max_retries + 1):
        try:
            print(f"    intento {attempt}/{max_retries}...")
            return fn(*args, **kwargs)
        except retry_on as exc:
            last_exc = exc
            delay_ms = compute_backoff_ms(attempt - 1, base_delay_ms)
            print(f"      fallo transitorio ({type(exc).__name__}): {exc}"
                  f" -- backoff modelado: {delay_ms}ms (no se duerme de verdad)")
    raise last_exc

Compare it against agent-fundamentals M7's call_with_retries: the for attempt in range(1, max_retries + 1), the except bounded to a specific exception type (never plain Exception — a validation error doesn't change by retrying it), and the final raise last_exc that never fakes a success that didn't happen, are exactly the same. The only genuinely new thing is compute_backoff_ms, and the line that uses it: instead of retrying immediately, every failure calculates — and shows — how long it should wait before the next attempt.

Honesty, before moving on: in a real system connected to a real network, that delay_ms would get used with time.sleep(delay_ms / 1000) (or its async equivalent) right before the next attempt. This guide never runs that wait — actually sleeping in every "What to expect" block would make every example slow and, worse, not byte-for-byte reproducible from one run to the next. What matters for learning the pattern — how the wait grows, and why that protects a saturated tool — comes across just as well in the calculated number as on the real clock. And a second honesty, the same one you already saw in Lesson 1: a real system almost always adds jitter — a random variation — to that number, so thousands of clients retrying the same dependency don't all do it at the exact same instant. This guide never implements jitter with random, for the same reproducibility reason; the complete version, with real random jitter and measured against thousands of simulated clients, is in resilience-and-reliability-patterns-guide (Module 3, "Retries, Backoff, and Jitter") — it's cited, not repeated.

Run for real: a brief blip, retry_with_backoff manages to recover

import reservo_tools as rt

_book_room_real = rt.book_room
_state = {"count": 0}


def flaky_book_room(room, tier, hours, member):
    """Falla las primeras DOS llamadas (un bache breve, no un apagón largo)
    y después funciona con normalidad."""
    _state["count"] += 1
    if _state["count"] <= 2:
        raise ConnectionError(f"timeout de red simulado (intento {_state['count']})")
    return _book_room_real(room, tier, hours, member)


print("=== max_retries=3, base_delay_ms=100: alcanza a recuperarse ===")
result = retry_with_backoff(
    flaky_book_room, room="Focus", tier="pro", hours=3, member="Ana",
    max_retries=3, base_delay_ms=100,
)
print("resultado:", result)

What to expect:

=== max_retries=3, base_delay_ms=100: alcanza a recuperarse ===
    intento 1/3...
      fallo transitorio (ConnectionError): timeout de red simulado (intento 1) -- backoff modelado: 100ms (no se duerme de verdad)
    intento 2/3...
      fallo transitorio (ConnectionError): timeout de red simulado (intento 2) -- backoff modelado: 200ms (no se duerme de verdad)
    intento 3/3...
resultado: {'booking_id': 1, 'confirmed': True, 'price_cents': 6000}

The first two attempts fail — exactly as flaky_book_room is designed to do — each with its backoff calculated and shown (100ms, then 200ms — double). The third attempt, with no failure message before resultado:, succeeded: _state["count"] already reached 3, and flaky_book_room runs the real branch. The complete table of how each backoff grows, with base_delay_ms=100:

print("=== la tabla de backoff exponencial (compute_backoff_ms), base=100ms ===")
for attempt in range(4):
    print(f"  intento {attempt}: delay = 100 * 2**{attempt} = {compute_backoff_ms(attempt, 100)}ms")

What to expect:

=== la tabla de backoff exponencial (compute_backoff_ms), base=100ms ===
  intento 0: delay = 100 * 2**0 = 100ms
  intento 1: delay = 100 * 2**1 = 200ms
  intento 2: delay = 100 * 2**2 = 400ms
  intento 3: delay = 100 * 2**3 = 800ms

Every round doubles the previous wait — the same progression, no matter which base_delay_ms you choose.

Run for real: max_retries=1 runs out before recovering

print("=== max_retries=1: se agota antes de recuperarse (el mismo bache) ===")
_state["count"] = 0
try:
    retry_with_backoff(
        flaky_book_room, room="Focus", tier="pro", hours=3, member="Ana",
        max_retries=1, base_delay_ms=100,
    )
except ConnectionError as exc:
    print(f"ConnectionError final, tras agotar el tope: {exc}")

What to expect:

=== max_retries=1: se agota antes de recuperarse (el mismo bache) ===
    intento 1/1...
      fallo transitorio (ConnectionError): timeout de red simulado (intento 1) -- backoff modelado: 100ms (no se duerme de verdad)
ConnectionError final, tras agotar el tope: timeout de red simulado (intento 1)

With only one chance, retry_with_backoff never reaches the third attempt — the one that would have worked — and re-raises the real ConnectionError, exactly like agent-fundamentals M7's call_with_retries did: it never hides a real failure behind a cap that's too tight.

Run for real: a long outage — even backoff isn't enough

_state["count"] = 0
OUTAGE_CALLS = 6

def flaky_book_room_long(room, tier, hours, member):
    _state["count"] += 1
    if _state["count"] <= OUTAGE_CALLS:
        raise ConnectionError(f"timeout de red simulado (intento {_state['count']})")
    return _book_room_real(room, tier, hours, member)

print("=== apagón largo (6 llamadas caídas): max_retries=3 con backoff NO alcanza ===")
try:
    retry_with_backoff(
        flaky_book_room_long, room="Focus", tier="pro", hours=3, member="Ana",
        max_retries=3, base_delay_ms=100,
    )
except ConnectionError as exc:
    print(f"ConnectionError final: {exc}")
print("llamadas reales gastadas en ESTE run: 3 de 3 -- las 3 tocaron la tool caída")

What to expect:

=== apagón largo (6 llamadas caídas): max_retries=3 con backoff NO alcanza ===
    intento 1/3...
      fallo transitorio (ConnectionError): timeout de red simulado (intento 1) -- backoff modelado: 100ms (no se duerme de verdad)
    intento 2/3...
      fallo transitorio (ConnectionError): timeout de red simulado (intento 2) -- backoff modelado: 200ms (no se duerme de verdad)
    intento 3/3...
      fallo transitorio (ConnectionError): timeout de red simulado (intento 3) -- backoff modelado: 400ms (no se duerme de verdad)
ConnectionError final: timeout de red simulado (intento 3)
llamadas reales gastadas en ESTE run: 3 de 3 -- las 3 tocaron la tool caída

This is the whole lesson's key moment: retry_with_backoff, with exponential backoff and everything, still can't recover from an outage that lasts longer than its own cap. And here's the problem Lesson 2 already showed, now with backoff in the mix but still unsolved: if the next user arrives a second later, with their own run, they're going to repeat exactly this same sequence of three failed attempts — because retry_with_backoff, just like call_with_retries, has no memory outside this single call. Backoff improves how long it takes to give up within a run; it doesn't stop the next run from trying it all over again from scratch.

Common mistakes

1. Using time.sleep(delay_ms / 1000) in one of this guide's "What to expect" blocks. It would break the byte-for-byte reproducibility every example demands, and would add real seconds of waiting to something that comes across just as well seeing the calculated number. In a real system connected to a real network, that sleep does belong — here, never.

2. Implementing jitter with random.uniform(...). It's the correct technique in production — it stops thousands of clients from all retrying at the exact same instant — but it introduces non-determinism into an example that has to produce the same output every time. If you need to illustrate jitter in your own exercise, use a fixed value or a deterministic sequence (for example, a predefined list of offsets), and be explicit that in production that value would be genuinely random.

3. Catching Exception instead of retry_on in retry_with_backoff. If the except were generic, a validation error — tier="premium", which is never going to change its result by being retried — would get retried just like a real network failure, wasting attempts (and, in a connected system, money) on something we already know isn't going to change. This is exactly the same warning from agent-fundamentals M7, Lesson 6.

4. Thinking a higher max_retries "solves" the last example's long outage. Raising the cap postpones the problem, it doesn't solve it: a real outage can last minutes, and no reasonable max_retries — without turning into an absurd wait within a single run — covers that. The right answer isn't "more retries within the run" — it's the memory across runs Lesson 4 builds.

Exercises

Exercise 1: Calculate the accumulated backoff (Easy)

Without running Python: with base_delay_ms=50 and three failed retries before a fourth successful attempt, how much backoff got calculated in total, adding up the three? (Remember: attempt N's backoff uses compute_backoff_ms(N - 1, base_delay_ms), because the first attempt, index 0, waits nothing before itself — backoff gets calculated after that attempt fails, before the next one).

Exercise 2: A tool that never recovers, with backoff (Medium)

Write always_down_book_room(**kwargs), which always raises ConnectionError("servicio permanentemente caído"), no matter how many times it's called. Run retry_with_backoff(always_down_book_room, max_retries=3, base_delay_ms=100) inside a try/except, and confirm the final ConnectionError gets re-raised after exactly three attempts, with backoff growing on each one.

def always_down_book_room(**kwargs):
    raise ConnectionError("servicio permanentemente caído")

try:
    retry_with_backoff(always_down_book_room, max_retries=3, base_delay_ms=100)
except ConnectionError as exc:
    print(f"ConnectionError final: {exc}")

    intento 1/3...
      fallo transitorio (ConnectionError): servicio permanentemente caído -- backoff modelado: 100ms (no se duerme de verdad)
    intento 2/3...
      fallo transitorio (ConnectionError): servicio permanentemente caído -- backoff modelado: 200ms (no se duerme de verdad)
    intento 3/3...
      fallo transitorio (ConnectionError): servicio permanentemente caído -- backoff modelado: 400ms (no se duerme de verdad)
ConnectionError final: servicio permanentemente caído

Exercise 3: Why shouldn't retry_with_backoff have a max_retries of 50? (Hard)

agent-fundamentals M7 (Lesson 6) already explained, for call_with_retries, why a huge cap "so I never miss a recovery" isn't free. Apply that same reasoning here, but now with exponential backoff in the mix: calculate how much accumulated backoff (the sum of every compute_backoff_ms) would get generated with max_retries=10 and base_delay_ms=100, never succeeding at all. Explain, in one or two sentences, why that number — even though it's never actually slept in this guide — is exactly the reason a real system should never set a max_retries that high without a circuit breaker involved.

total_ms = sum(compute_backoff_ms(attempt, 100) for attempt in range(10))
print(f"backoff acumulado con max_retries=10: {total_ms}ms")

backoff acumulado con max_retries=10: 102300ms

Summary and next step

• retry_with_backoff(fn, *args, max_retries=3, base_delay_ms=100, retry_on=(ConnectionError,), **kwargs) scales up agent-fundamentals M7's call_with_retries: same criteria for what gets retried and what doesn't, same hard cap, with the piece that lesson left pending — exponential backoff (delay = base * 2**attempt), calculated and shown, never actually slept.
• We ran it with flaky_book_room: a brief blip recovers within the cap, with backoff growing on every attempt (100ms, 200ms, 400ms...); a max_retries=1 runs out early; and a long outage (6 downed calls) demonstrates that even backoff isn't enough when the outage lasts longer than the run's cap.
• We confirmed, with numbers, why a huge max_retries isn't the solution: accumulated backoff grows exponentially, and even so no cap within a run solves the underlying problem — the lack of memory across runs.
• Explicit honesty: this guide never actually sleeps backoff (time.sleep()) or uses random for jitter — the pattern comes across just as well calculated; the version with real random jitter is in resilience-and-reliability-patterns-guide.

Next lesson: 04 — The Circuit Breaker Pattern. We build the memory everything seen so far is missing: an object that remembers, across runs, that a tool has been failing, and stops calling it until a periodic check confirms it came back.

Additional resources

1. Python — Built-in exceptions (ConnectionError) — The standard exception reused from agent-fundamentals M7 to represent a transient failure.
2. Anthropic — Errors — Reference for Claude API error codes; a real guide to which failure types tend to be transient (for example, 529 overloaded_error) versus which aren't — the same distinction retry_on encodes here.
3. resilience-and-reliability-patterns-guide (Module 3, "Retries, Backoff, and Jitter") — The complete version of exponential backoff, with real random jitter, measured in depth against a retry storm of thousands of simulated clients — the depth this module cites instead of repeating.
4. Python 3.14 — What's New — The version every line of code in this lesson ran on.