Retry strategies and idempotency

Overview

Circuit breakers protect you from a downstream that's completely broken. But most failures are transient: a 200ms network blip, a 429 that resolves in seconds, a 503 that passes with the next request. For those cases, retry is the right tool — only if you do it well.

Retry done badly is worse than no retry: aggressive and unbounded it burns money (each retry to the LLM costs tokens), without idempotency it duplicates side-effects (sends the email twice, charges twice), without backoff synchronized across instances it amplifies load spikes.

In this lesson you're going to learn the complete set: when to retry, when not to, how to do it with backoff that doesn't hammer, idempotency keys that avoid duplicates, and dead letter queues that serve as diagnostics, not as a graveyard.

By the end you'll be able to:

• Decide when to retry (transient errors) vs when not to (permanent errors)
• Implement exponential backoff with correct jitter
• Design idempotency keys that survive retries
• Configure dead letter queues with automated analysis

---

When to retry (when not to)

Simple rule: retry only if the error is transient.

Error	Transient	Action
Connection reset, connection refused	✅	Retry with backoff
Timeout	✅	Retry with backoff
HTTP 502/503/504	✅	Retry with backoff
HTTP 429 (rate limit)	✅	Retry following Retry-After
HTTP 500	⚠️ Depends	Retry once; if it persists, don't insist
HTTP 4xx (except 429)	❌	NO retry (invalid request)
Auth error (401/403)	❌	NO retry (bad config)
Validation error	❌	NO retry (bad input)
Content filter (OpenAI)	❌	NO retry (unacceptable prompt)

For AI: if the LLM responds with content == "" or finish_reason == "content_filter", NO retry. It's the model's correct response.

---

Exponential backoff: why exponential

If your service retries every 100ms, 10 instances × 10 retries/s = 100 requests/s toward the provider that's already down. You amplify the problem.

Exponential backoff: each retry waits double the previous one.

attempt 1: wait 1s
attempt 2: wait 2s
attempt 3: wait 4s
attempt 4: wait 8s

This gives the downstream time to recover. For LLM providers, up to 60s of total wait is reasonable.

Basic implementation

import time
import random

def with_exponential_backoff(func, max_retries=3, base_delay=1.0, max_delay=60.0):
    for attempt in range(max_retries + 1):
        try:
            return func()
        except RetriableError as e:
            if attempt == max_retries:
                raise
            delay = min(max_delay, base_delay * (2 ** attempt))
            time.sleep(delay)

---

Jitter: the crucial detail

Without jitter, all your instances retry at the same time:

t=0     : 10 instances try, all fail
t=1s    : 10 instances retry at the same time → second spike
t=3s    : 10 instances retry at the same time → third spike

This is called "thundering herd". You avoid it with jitter: adding randomness to each delay.

def with_jittered_backoff(func, max_retries=3, base_delay=1.0, max_delay=60.0):
    for attempt in range(max_retries + 1):
        try:
            return func()
        except RetriableError:
            if attempt == max_retries:
                raise
            # Full jitter: delay ∈ [0, base * 2^attempt]
            delay = random.uniform(0, min(max_delay, base_delay * (2 ** attempt)))
            time.sleep(delay)

There are variants (decorrelated jitter, equal jitter) — for 95% of cases, full jitter is what you want.

---

Respecting Retry-After

When a provider returns 429 with Retry-After: 30, it's telling you how long to wait. Respect it. If you retry before that, you receive 429 again and worse: the provider could extend your cool-down.

import httpx

def call_with_retry_after(url: str, max_retries=3):
    for attempt in range(max_retries + 1):
        r = httpx.get(url)
        if r.status_code == 429 and attempt < max_retries:
            retry_after = float(r.headers.get("Retry-After", "1"))
            time.sleep(retry_after + random.uniform(0, 1))  # + jitter
            continue
        return r

---

When to combine retry with a circuit breaker

Retry and circuit breaker aren't alternatives — they're complementary.

Recommended pattern:

def call_protected(prompt: str):
    # Outer layer: circuit breaker
    return circuit.call(
        lambda: with_retry(  # Inner layer: retry
            lambda: openai_client.chat.completions.create(...)
        )
    )

• The circuit breaker decides "should I try the provider at all?"
• Retry decides "this specific request failed, do I retry or give up?"

When the circuit is OPEN, retry doesn't run (immediate rejection → fallback). When the circuit is CLOSED, retry handles the transient blips without opening the circuit unnecessarily.

---

Idempotency: the problem and the solution

Problem: if you retry a request that already partially executed, you can duplicate side-effects.

Examples of side-effects in AI:

• Sending an email to the user with the LLM's response (one retry = two emails)
• Charging the customer for tokens used (one retry = double charge)
• Logging the request for auditing (one retry = two logs, inflated metrics)
• Storing the result in the DB (one retry = two records, possibly with conflicts)

Solution: idempotency keys. Each request carries a unique ID. Your system checks whether it already processed that ID before doing side-effects.

Basic pattern

import redis

r = redis.Redis()

def process_with_idempotency(idempotency_key: str, work):
    # 1. Try to mark the key as "in progress"
    acquired = r.set(
        f"idem:{idempotency_key}",
        "in_progress",
        nx=True,  # only set if it doesn't exist
        ex=300,   # expiry 5min
    )

    if not acquired:
        # It already exists — return the previous result or wait
        existing = r.get(f"idem:{idempotency_key}:result")
        if existing:
            return existing
        raise Exception("Request in progress, try again in a few seconds")

    # 2. Process
    try:
        result = work()
        r.set(f"idem:{idempotency_key}:result", result, ex=3600)
        r.delete(f"idem:{idempotency_key}")  # removes the "in progress"
        return result
    except Exception:
        r.delete(f"idem:{idempotency_key}")  # releases for retry
        raise

In your API

@app.post("/api/chat")
def chat(req: ChatRequest, idempotency_key: str | None = Header(None)):
    key = idempotency_key or str(uuid4())
    result = process_with_idempotency(key, lambda: call_llm(req.prompt))
    return {"text": result, "idempotency_key": key}

The client can pass Idempotency-Key: <uuid> in the header. If the request goes without a response (network blip), it retries with the same key and your system returns the original result without reprocessing.

---

Idempotency for queue-based processing

In the flow of M5-03 (queue + worker), you have two places where there can be duplication:

1. The API enqueues twice (the client retries before receiving the 202)
2. The worker processes twice (visibility timeout, restart)

Solution for (1): idempotency at enqueue

@app.post("/api/chat", status_code=202)
def enqueue_chat(req: ChatRequest, idempotency_key: str | None = Header(None)):
    key = idempotency_key or str(uuid4())

    # Check if we already enqueued this key
    existing_job = r.hget("idem_to_job", key)
    if existing_job:
        return {"job_id": existing_job.decode(), "status": "queued"}

    job_id = str(uuid4())
    r.hset("idem_to_job", key, job_id)
    r.expire("idem_to_job", 3600)
    r.lpush("llm-jobs", json.dumps({...}))

    return {"job_id": job_id, "status": "queued"}

Solution for (2): the worker checks before processing

def worker_loop():
    while True:
        payload = dequeue()
        if not payload:
            continue

        # Check if we already processed it
        existing = r.hget("job_results", payload["job_id"])
        if existing:
            existing_data = json.loads(existing)
            if existing_data.get("status") in ("completed", "failed"):
                continue  # already processed, ignore

        # Mark as "processing" atomically
        was_set = r.hsetnx("job_results", payload["job_id"],
                            json.dumps({"status": "processing", "worker_id": WORKER_ID}))
        if not was_set:
            continue  # another worker took this job

        try:
            process(payload)
        except Exception as e:
            # ...

---

Dead letter queues: diagnostics, not a graveyard

When a job fails N times (e.g., 5), you don't want to retry it infinitely. You send it to a dead letter queue (DLQ). But the DLQ shouldn't be a black hole.

Correct pattern

MAX_RETRIES = 5

def worker_with_dlq():
    while True:
        payload = dequeue()
        attempts = payload.get("attempts", 0)

        try:
            result = process(payload)
            store_result(payload["job_id"], result)
        except Exception as e:
            attempts += 1
            if attempts >= MAX_RETRIES:
                # → DLQ
                r.lpush("dlq:llm-jobs", json.dumps({
                    **payload,
                    "final_error": str(e),
                    "final_traceback": traceback.format_exc(),
                    "moved_to_dlq_at": time.time(),
                    "attempts": attempts,
                }))
                # ALERT
                alert_team(f"Job {payload['job_id']} went to DLQ after {attempts} attempts")
            else:
                # Re-enqueue with backoff
                delay = min(60, 2 ** attempts)
                time.sleep(delay)
                payload["attempts"] = attempts
                r.lpush("llm-jobs", json.dumps(payload))

Processing the DLQ

A DLQ with 1000 jobs untouched = a problem. A DLQ must be operated:

1. Alert when it grows (>10 items, or growth >X/hr)
2. Dashboard showing jobs in the DLQ with categorization (by error type)
3. Manual replay after a fix: take jobs from the DLQ and re-enqueue them to the main queue
4. Automated analysis: group by similar root cause, expose patterns

# Simple replay
def replay_dlq():
    while True:
        item = r.brpop("dlq:llm-jobs", timeout=5)
        if not item:
            break
        payload = json.loads(item[1])
        payload["attempts"] = 0  # reset counter
        r.lpush("llm-jobs", json.dumps(payload))
        print(f"Replayed {payload['job_id']}")

---

Common traps

Trap 1 — Retrying a 400. "Bad Request" means the client sent something invalid. Retry doesn't fix it. Only retry for 5xx, 429, network errors, timeouts.

Trap 2 — Backoff without jitter. 10 instances retry simultaneously, your provider goes down harder. Always jitter.

Trap 3 — Infinite max retries. Without a limit, a poisoned job can run forever. Always a finite max_retries (3-5 typically).

Trap 4 — Idempotency key generated on the client without guarantees. If the client generates a new UUID on each retry, it's not idempotent. The idempotency key must be stable across retries from the same client.

Trap 5 — DLQ without alert or monitoring. Jobs go to the DLQ silently. You arrive Monday to 5000 dead jobs. Alerts are mandatory.

Trap 6 — Retry in a circle between services. Service A calls B with retry. B calls C with retry. C fails. Your total retry: 3 × 3 × 3 = 27 requests for each original request. Don't have uncoordinated retry in a chain. Retry in a single layer, ideally the one closest to the downstream.

---

Exercise

Your system processes analysis requests with an LLM. Each request involves:

1. Calling the LLM with the user's content (can fail transiently)
2. Storing the result in the DB
3. Sending an email to the user with the result

Design retry + idempotency:

1. For which of the 3 steps do you apply retry? With what parameters?
2. How do you prevent a retry from sending the email twice?
3. What happens if the LLM call works but the email fails?
4. When does it go to the DLQ?

See solution

1. Retry per step:
   • LLM call: retry 3 times with exponential backoff + full jitter, base 2s, max 60s. NO retry on 4xx, content_filter, empty content.
   • DB save: retry 3 times with short backoff (100ms-1s). Almost always transient (brief timeouts).
   • Email: retry 3 times with backoff (1s-10s). 4xx from the email provider NO retry; 5xx and timeouts yes.
2. Idempotency:
   • The job has a unique job_id
   • Before sending the email, mark it in the DB: UPDATE jobs SET email_sent_at = NOW() WHERE id = ? AND email_sent_at IS NULL — only proceed if rowcount == 1
   • If a retry tries to send it again, the rowcount is 0 and skip
3. LLM OK but email fails:
   • The DB has the result (step 2 OK)
   • The email has status "failed", email_attempts = 3
   • The overall job marks partial_success: the user can see the result in the app even though they didn't receive the email
   • Alert support to investigate the email
4. DLQ:
   • The LLM call fails 3 times in a row (with backoff between each)
   • After 3 attempts, the job goes to the DLQ with the last error
   • The oncall team sees the alert, investigates (malicious prompt? OpenAI outage? bug?), decides to replay or discard

---

Summary

You learned:

• ✅ When to retry (transient) vs when not to (4xx, content_filter, validation)
• ✅ Exponential backoff with full jitter (avoids thundering herd)
• ✅ Respecting Retry-After when the provider sends it
• ✅ Combining retry (individual request) with a circuit breaker (whole downstream)
• ✅ Idempotency keys: stable across retries, stored in Redis with expiry
• ✅ Idempotency in queue + worker: dedup at enqueue and at processing
• ✅ Dead letter queues with alerts, dashboards, manual replay

Checkpoint: if you can design retry for a pipeline that has an LLM call + DB + email, without duplicating side-effects, you're ready.

---

Next lesson

07 — Graceful degradation by levels. Your system already handles errors, latency, rate limits, retry, circuit breakers. But when everything happens at once, how do you respond? We're going to design levels of degradation that keep the system partially functional even in extreme conditions.

---

Resources

1. Exponential Backoff and Jitter — AWS Blog — the canonical write-up on jitter.
2. Tenacity (Python) — production-ready retry library.
3. Stripe API — Idempotency — reference for how Stripe does it.
4. Designing for Failure (Adrian Cockcroft) — Netflix patterns.
5. AWS SQS — Visibility timeout — for idempotency in queues.