3. Retry with Exponential Backoff

Description

Retry without a strategy is worse than no retry: if OpenAI is saturated and your app sends 100 simultaneous requests that fail, retrying them immediately only makes the problem worse. Exponential backoff + jitter is the strategy that turns the "thundering herd" problem into a controlled distribution of retries. In this capsule you'll implement complete retry with tenacity, including logging callbacks, support for the Retry-After header, and tests that verify the behavior without depending on real delays.

The naive retry problem

# ❌ Immediate retry — the anti-pattern

def call_llm_naive(prompt: str) -> str:
    for attempt in range(3):
        try:
            return client.chat.completions.create(...)
        except RateLimitError:
            if attempt < 2:
                continue  # ← IMMEDIATE retry
            raise

# Scenario: 50 users make requests at the same time
# OpenAI returns 429 for all of them
# They all do an IMMEDIATE retry
# OpenAI receives 150 requests (50 × 3) in milliseconds
# They all fail with 429 again
# The retries make the problem worse
# Result: all users see error 500

# ─────────────────────────────────────────────────────────────

# ✅ Exponential backoff + jitter — the solution

# Scenario: 50 users make requests at the same time
# They all receive 429
# Retry 1: wait 1.0s + random(0, 0.5)s → different moments
# Retry 2: wait 2.0s + random(0, 1.0)s → distributed
# Retry 3: wait 4.0s + random(0, 2.0)s → well distributed
# The 50 retries spread out over ~10 seconds instead of milliseconds
# OpenAI processes gradually, most complete

tenacity: the retry library

pip install tenacity

# The building blocks of tenacity

from tenacity import (
    retry,                    # Main decorator
    stop_after_attempt,       # Maximum N attempts
    stop_after_delay,         # Maximum N total seconds
    wait_exponential,         # Exponential wait: 1s, 2s, 4s, 8s...
    wait_random_exponential,  # Exponential + jitter: 1.2s, 2.7s, 4.1s...
    wait_fixed,               # Fixed wait: always N seconds
    retry_if_exception_type,  # Retry only for certain exception types
    retry_if_exception,       # Retry if the function returns True for the error
    before_sleep,             # Callback BEFORE sleeping (logging)
    after,                    # Callback AFTER each attempt
    RetryError,               # Exception if all attempts fail
)

Basic configuration with tenacity

# src/infrastructure/retry_config.py
from openai import APITimeoutError, RateLimitError, APIConnectionError, InternalServerError
from tenacity import retry, stop_after_attempt, wait_random_exponential, retry_if_exception

def is_retryable_openai_error(exception: Exception) -> bool:
    """
    Determine whether an OpenAI exception is worth retrying.
    
    Transients → retry:
    - APITimeoutError: the model took long, it may work again
    - RateLimitError: the limit frees up over time
    - APIConnectionError: the network failed momentarily
    - InternalServerError (5xx): server error, it may recover
    
    Permanent → no retry:
    - AuthenticationError (401): the key is wrong, retry doesn't help
    - BadRequestError (400): the input is wrong, retry will give the same error
    - PermissionDeniedError (403): no access, retry doesn't help
    """
    from openai import AuthenticationError, BadRequestError, PermissionDeniedError
    
    # No retry for permanent errors
    if isinstance(exception, (AuthenticationError, BadRequestError, PermissionDeniedError)):
        return False
    
    # Retry for known transients
    if isinstance(exception, (APITimeoutError, RateLimitError,
                              APIConnectionError, InternalServerError)):
        return True
    
    # For LLMProviderError, use the integrated classification
    from src.infrastructure.llm_provider import LLMProviderError
    if isinstance(exception, LLMProviderError):
        return exception.should_retry
    
    # For unknown errors, no retry by default (safe)
    return False


# Standard configuration for most AI apps
STANDARD_RETRY = retry(
    # Stop after 4 attempts (1 original + 3 retries)
    stop=stop_after_attempt(4),
    
    # Exponential backoff with jitter:
    # Attempt 1: 1s base + random(0, 1)s
    # Attempt 2: 2s base + random(0, 1)s
    # Attempt 3: 4s base + random(0, 2)s
    # Maximum 30s wait
    wait=wait_random_exponential(multiplier=1, min=1, max=30),
    
    # Only retry transient errors
    retry=retry_if_exception(is_retryable_openai_error),
    
    # If all attempts fail, propagate the original exception
    reraise=True
)

# More aggressive configuration for critical operations
AGGRESSIVE_RETRY = retry(
    stop=stop_after_attempt(5),
    wait=wait_random_exponential(multiplier=2, min=2, max=60),
    retry=retry_if_exception(is_retryable_openai_error),
    reraise=True
)

# Light configuration for low-priority operations
LIGHT_RETRY = retry(
    stop=stop_after_attempt(2),  # Only 1 retry
    wait=wait_random_exponential(multiplier=1, min=1, max=10),
    retry=retry_if_exception(is_retryable_openai_error),
    reraise=True
)

RetryProvider: wrapping the LLMProvider

# src/infrastructure/retry_provider.py
import structlog
from tenacity import retry, RetryError, before_sleep_log
from src.infrastructure.llm_provider import LLMProvider, LLMProviderError
from src.infrastructure.retry_config import is_retryable_openai_error, STANDARD_RETRY

log = structlog.get_logger()

class RetryProvider:
    """
    Wrapper that adds retry with exponential backoff to any LLMProvider.
    
    Implements the LLMProvider Protocol, so it's completely
    transparent to the code that uses it.
    """
    
    def __init__(
        self,
        inner: LLMProvider,
        max_attempts: int = 4,
        min_wait_seconds: float = 1.0,
        max_wait_seconds: float = 30.0
    ):
        self._inner = inner
        self._max_attempts = max_attempts
        self._min_wait = min_wait_seconds
        self._max_wait = max_wait_seconds
        self._attempt_count = 0  # For metrics
        self._retry_count = 0
    
    def complete(self, messages: list[dict], **kwargs) -> str:
        """
        Call the inner provider with automatic retry for transient errors.
        """
        self._attempt_count += 1
        attempt_number = 0
        last_error = None
        
        # Create the retry decorator dynamically with the configured parameters
        from tenacity import (
            retry as tenacity_retry, stop_after_attempt,
            wait_random_exponential, retry_if_exception
        )
        
        @tenacity_retry(
            stop=stop_after_attempt(self._max_attempts),
            wait=wait_random_exponential(
                multiplier=1,
                min=self._min_wait,
                max=self._max_wait
            ),
            retry=retry_if_exception(is_retryable_openai_error),
            reraise=True,
            before_sleep=self._log_retry_attempt
        )
        def _call_with_retry():
            return self._inner.complete(messages, **kwargs)
        
        return _call_with_retry()
    
    def _log_retry_attempt(self, retry_state) -> None:
        """Callback that runs before each sleep between retries."""
        self._retry_count += 1
        
        log.warning(
            "llm_retry_attempt",
            attempt_number=retry_state.attempt_number,
            exception_type=type(retry_state.outcome.exception()).__name__,
            sleep_seconds=round(retry_state.next_action.sleep, 1),
            total_retries=self._retry_count
        )
    
    @property
    def total_retries(self) -> int:
        """Total retries performed (useful for metrics)."""
        return self._retry_count

Using tenacity as a decorator directly

# If you prefer a decorator instead of a wrapper class:

from openai import APITimeoutError, RateLimitError, APIConnectionError
from tenacity import retry, stop_after_attempt, wait_random_exponential, retry_if_exception_type
import structlog

log = structlog.get_logger()

def log_retry(retry_state):
    log.warning(
        "llm_retry",
        attempt=retry_state.attempt_number,
        error=str(retry_state.outcome.exception())[:100],
        wait_seconds=round(getattr(retry_state.next_action, 'sleep', 0), 1)
    )

@retry(
    stop=stop_after_attempt(4),
    wait=wait_random_exponential(multiplier=1, min=1, max=30),
    retry=retry_if_exception_type((APITimeoutError, RateLimitError, APIConnectionError)),
    reraise=True,
    before_sleep=log_retry
)
def call_openai_with_retry(client, messages: list[dict], model: str,
                            temperature: float, max_tokens: int) -> str:
    response = client.chat.completions.create(
        model=model,
        messages=messages,
        temperature=temperature,
        max_tokens=max_tokens
    )
    return response.choices[0].message.content

The jitter effect visualized

# Why jitter? Visualizing the problem:

# WITHOUT jitter — thundering herd:
# t=0:   50 requests fail
# t=1s:  50 simultaneous retries fail
# t=2s:  50 simultaneous retries fail
# t=4s:  50 simultaneous retries fail
# All retries collide at the same moments

# WITH jitter — natural distribution:
# t=0:     50 requests fail
# t=1.0s:  3 retries
# t=1.1s:  7 retries
# t=1.3s:  12 retries
# t=1.6s:  15 retries
# t=2.0s:  8 retries
# t=2.1s:  5 retries
# The retries spread out, reducing pressure on the API

import random
import time

def exponential_backoff_with_jitter(attempt: int, base: float = 1.0, max_wait: float = 30.0) -> float:
    """Calculate the wait time with jitter."""
    exponential = base * (2 ** attempt)  # 1, 2, 4, 8, 16...
    with_cap = min(exponential, max_wait)  # No more than max_wait
    with_jitter = with_cap * (0.5 + random.random() * 0.5)  # ±50% random
    return with_jitter

# Example of generated times for 3 retries:
for attempt in range(3):
    wait_time = exponential_backoff_with_jitter(attempt)
    print(f"Attempt {attempt+1}: wait {wait_time:.2f}s")
# Attempt 1: wait 0.73s
# Attempt 2: wait 1.87s
# Attempt 3: wait 3.14s
# (different each time due to the random)

Respecting the Retry-After header

# When OpenAI returns 429, it sometimes includes Retry-After
# We should respect that time instead of our backoff

from tenacity import wait_base
import time

class WaitWithRetryAfter(wait_base):
    """
    Custom wait strategy for tenacity that respects the Retry-After header.
    If the error has a retry_after, it waits that time.
    Otherwise, it uses exponential backoff with jitter.
    """
    def __init__(self, multiplier: float = 1.0, min_wait: float = 1.0, max_wait: float = 60.0):
        self._multiplier = multiplier
        self._min = min_wait
        self._max = max_wait
    
    def __call__(self, retry_state) -> float:
        exc = retry_state.outcome.exception()
        
        # Try to get the Retry-After from the error
        if hasattr(exc, "retry_after") and exc.retry_after:
            return min(exc.retry_after, self._max)
        
        # Also try in LLMProviderError
        from src.infrastructure.llm_provider import LLMProviderError
        if isinstance(exc, LLMProviderError) and exc.retry_after:
            return min(exc.retry_after, self._max)
        
        # Fallback to exponential backoff
        attempt = retry_state.attempt_number - 1  # 0-indexed
        exponential = self._multiplier * (2 ** attempt)
        with_cap = min(exponential, self._max)
        return max(self._min, with_cap * (0.5 + 0.5 * __import__("random").random()))

# Usage:
@retry(
    stop=stop_after_attempt(4),
    wait=WaitWithRetryAfter(multiplier=1.0, min_wait=1.0, max_wait=60.0),
    retry=retry_if_exception(is_retryable_openai_error),
    reraise=True
)
def call_with_retry_after_support(messages):
    return client.chat.completions.create(...)

Retry tests

# tests/unit/test_retry_provider.py
import pytest
from unittest.mock import MagicMock, patch
from openai import APITimeoutError, AuthenticationError
from src.infrastructure.retry_provider import RetryProvider
from src.infrastructure.mock_provider import MockProvider
from src.infrastructure.llm_provider import LLMProviderError
from src.infrastructure.error_classifier import ErrorCategory

class TestRetryProvider:
    
    def test_succeeds_on_first_attempt_without_retry(self):
        """If the first call works, there are no retries."""
        inner = MockProvider('{"sentiment": "positive", "score": 0.8, "confidence": 0.9}')
        retry_provider = RetryProvider(inner, max_attempts=3)
        
        result = retry_provider.complete([{"role": "user", "content": "test"}])
        
        assert '{"sentiment"' in result
        assert inner.call_count == 1
        assert retry_provider.total_retries == 0
    
    def test_retries_on_transient_error_then_succeeds(self):
        """If it fails 2 times and then works, it must have 3 calls."""
        transient_error = LLMProviderError(
            "Timeout",
            category=ErrorCategory.TRANSIENT,
            should_retry=True
        )
        inner = MockProvider(
            responses=["error", "error", '{"sentiment": "positive", "score": 0.8, "confidence": 0.9}'],
            raise_error=None
        )
        
        call_count = 0
        original_complete = inner.complete
        
        def tracked_complete(messages, **kwargs):
            nonlocal call_count
            call_count += 1
            if call_count <= 2:
                raise transient_error
            return original_complete(messages, **kwargs)
        
        inner.complete = tracked_complete
        
        retry_provider = RetryProvider(inner, max_attempts=4, min_wait_seconds=0.01, max_wait_seconds=0.1)
        result = retry_provider.complete([{"role": "user", "content": "test"}])
        
        assert call_count == 3
    
    def test_reraises_after_max_attempts(self):
        """If it fails max_attempts times, it must propagate the exception."""
        error = LLMProviderError("Rate limit", category=ErrorCategory.TRANSIENT, should_retry=True)
        
        call_count = 0
        def failing_complete(messages, **kwargs):
            nonlocal call_count
            call_count += 1
            raise error
        
        inner = MockProvider()
        inner.complete = failing_complete
        
        retry_provider = RetryProvider(inner, max_attempts=3, min_wait_seconds=0.01, max_wait_seconds=0.1)
        
        with pytest.raises(LLMProviderError):
            retry_provider.complete([{"role": "user", "content": "test"}])
        
        assert call_count == 3  # Exactly max_attempts attempts
    
    def test_does_not_retry_non_retryable_error(self):
        """For non-retryable errors, it only tries once."""
        error = LLMProviderError(
            "Invalid API key",
            category=ErrorCategory.AUTH_ERROR,
            should_retry=False
        )
        
        call_count = 0
        def auth_failing(messages, **kwargs):
            nonlocal call_count
            call_count += 1
            raise error
        
        inner = MockProvider()
        inner.complete = auth_failing
        
        retry_provider = RetryProvider(inner, max_attempts=3, min_wait_seconds=0.01, max_wait_seconds=0.1)
        
        with pytest.raises(LLMProviderError):
            retry_provider.complete([{"role": "user", "content": "test"}])
        
        assert call_count == 1  # Only 1 attempt — no retry for auth errors

Exercises

Exercise 1: Calculate wait times

For a configuration with wait_exponential(multiplier=1, min=1, max=30), what would the wait times between retries be? (without jitter)

Exercise 2: Add logging to the retry

Modify the tenacity configuration so it logs each attempt with: attempt number, error type, and wait seconds:

import structlog
log = structlog.get_logger()

def log_before_retry(retry_state):
    log.warning(
        "llm_retry_attempt",
        attempt=retry_state.attempt_number,
        error_type=type(retry_state.outcome.exception()).__name__,
        sleep_seconds=round(getattr(retry_state.next_action, 'sleep', 0), 1)
    )

@retry(
    stop=stop_after_attempt(4),
    wait=wait_random_exponential(min=1, max=30),
    retry=retry_if_exception(is_retryable_openai_error),
    reraise=True,
    before_sleep=log_before_retry  # ← callback
)
def my_llm_call(messages):
    ...

Exercise 3: Configure retry for different scenarios

Write the tenacity configuration for these two cases:

1. Payment operation — if it fails, the user loses the transaction. You need to be aggressive with retries.
2. Cache preload — if it fails, nothing serious happens. It can retry once and move on.

# 1. Payment operation (aggressive)
PAYMENT_RETRY = retry(
    stop=stop_after_attempt(5),
    wait=wait_random_exponential(multiplier=2, min=2, max=60),
    retry=retry_if_exception(is_retryable_openai_error),
    reraise=True
)

# 2. Cache preload (light)
CACHE_RETRY = retry(
    stop=stop_after_attempt(2),
    wait=wait_random_exponential(multiplier=1, min=1, max=10),
    retry=retry_if_exception(is_retryable_openai_error),
    reraise=True
)

Exercise 4: Write a retry test without real delays

Write a test that verifies that RetryProvider makes exactly 3 attempts when the inner provider fails with a transient error, without waiting for the real delays:

def test_retries_exactly_max_attempts():
    """Use low min_wait and max_wait so the tests are fast."""
    call_count = 0
    error = LLMProviderError(
        "Timeout", category=ErrorCategory.TRANSIENT, should_retry=True
    )
    
    def always_fail(messages, **kwargs):
        nonlocal call_count
        call_count += 1
        raise error
    
    inner = MockProvider()
    inner.complete = always_fail
    
    # min_wait=0.01 and max_wait=0.01 eliminate the real delays
    provider = RetryProvider(inner, max_attempts=3,
                            min_wait_seconds=0.01, max_wait_seconds=0.01)
    
    with pytest.raises(LLMProviderError):
        provider.complete([{"role": "user", "content": "test"}])
    
    assert call_count == 3

Troubleshooting

"My retries take too long in tests"

Set min_wait_seconds=0.01 and max_wait_seconds=0.1 in your tests. Never use the production values (1-30s) in unit tests — they would make your suite take minutes.

"tenacity doesn't retry my error"

Verify that your is_retryable_openai_error function returns True for that error. The most common mistake is passing a wrapped exception: if RetryProvider receives an LLMProviderError but is_retryable_openai_error only checks OpenAI's exceptions directly, it won't match. Make sure it also checks LLMProviderError.should_retry.

"I get RetryError instead of the original exception"

If you use reraise=False (tenacity's default), the exception you get is tenacity.RetryError, not the original one. Always use reraise=True to propagate the original exception after exhausting the attempts.

"How do I know if the retry is working in production?"

Look for the llm_retry_attempt event in your logs. If you never see it, either your system works well and there are no transient errors, or your before_sleep callback isn't configured. If you see it too often, your API may have a persistent problem and you need a circuit breaker (capsule 04).

Summary

• Retry without backoff is harmful: it aggravates the rate limiting problem
• Exponential backoff: each attempt waits twice as long as the previous
• Jitter: adds randomness to avoid the thundering herd
• Only transient errors: no retry for auth errors, bad requests, etc.
• Retry-After: if the API indicates when to retry, respect it
• RetryProvider wrapper: transparent to the domain thanks to M6's DI
• 3-4 attempts: more attempts = more latency perceived by the user

Additional resources

1. tenacity Documentation — The complete library
2. Exponential Backoff and Jitter (AWS) — The canonical article on jitter
3. OpenAI Rate Limits — Official retry guide
4. Thundering Herd Problem — The problem that jitter solves