Technical and budget rate limiting

Overview

Your queue processes async. Your system responds fast to the user. But a client with a bug can flood your queue with 10,000 requests/minute, drain your token budget in 3 hours, and trigger LLM provider rate limits that affect all your legitimate users. Rate limiting is what prevents that scenario.

For traditional web systems, rate limiting is mainly technical protection: limiting requests per IP to avoid saturating the database. For AI, rate limiting is doubly important:

• Technical protection: don't saturate your system or the LLM provider's
• Budget protection: each request costs money, without a limit you run out of budget

In this lesson you're going to design rate limiting that considers both dimensions, with different granularities (per user, per tenant, global), using the right algorithms for AI.

By the end you'll be able to:

• Choose the right algorithm (token bucket vs leaky bucket vs fixed window)
• Design multi-level rate limits: user → tenant → global
• Implement rate limits that count tokens and dollars, not just requests
• Configure the behavior when the limit is reached: hard reject, queue, degradation
• Coordinate your rate limit with the LLM provider's so they don't contradict each other

---

Algorithms on one page

Token Bucket

Each user has a "bucket" of tokens. Each request consumes tokens. The tokens refill at a constant rate.

• Refill: 10 tokens/second
• Capacity: 100 tokens
• Each request: consumes 1 token (or N if the request is more expensive)

Pros: allows bursts (a client that saves up can spend all at once), simple to reason about. Cons: a distributed implementation requires synchronization (Redis with Lua scripts).

Recommended for AI: the most used algorithm, especially because it lets you express "this request costs more" (consume N tokens based on the LLM's tokens, GPU time, etc.).

Leaky Bucket

Requests enter a "bucket", leave at a constant rate. If they enter faster than they leave, the bucket fills and the excess is discarded.

Pros: smooths bursts (uniform output), good for protecting downstreams. Cons: less flexible, requests "wait" in the bucket.

When to use it: when you protect a downstream that doesn't tolerate bursts (e.g., an LLM provider with a hard rate limit).

Fixed Window

Counts requests in fixed windows (each minute, each hour). Reset at the window change.

Pros: trivial to implement. Cons: "edge effect" — two bursts adjacent at the window boundary consume 2× the real limit.

When to use it: prototypes, cases where edge effects don't matter.

Sliding Window

Counts requests in a moving window (the last 60s, not "this minute").

Pros: no edge effect. Cons: more complex to implement (you need request timestamps).

When to use it: when the fixed window's edge effect is a problem (high cardinality of clients with bursts).

Default for AI: Token Bucket with variable cost.

---

Levels of rate limiting

Rate limiting must be applied at multiple levels simultaneously:

Level	Example	Reason
Per user	60 req/hr	Fairness between users; one user can't saturate
Per tenant / organization	1000 req/hr	Pricing plan; tenants pay for a tier
Per endpoint	/api/chat: 60/min, /api/embed: 600/min	Expensive endpoints have stricter limits
System-wide	10K req/min	Absolute capacity protection
Vendor (downstream)	OpenAI: 60 RPM, 90K TPM	Respect the provider's limits

Rule: each request must pass all the applicable levels. If any level rejects it, the request is rejected.

---

Token bucket with variable cost: implementation

What matters for AI is that not all requests cost the same. A chat with max_tokens=50 costs less than one with max_tokens=2000. Your rate limit must reflect that.

# rate_limiter.py
import time
import redis

r = redis.Redis()

LUA_TOKEN_BUCKET = """
local key = KEYS[1]
local rate = tonumber(ARGV[1])
local capacity = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local cost = tonumber(ARGV[4])

local bucket = redis.call('hmget', key, 'tokens', 'last_refill')
local tokens = tonumber(bucket[1]) or capacity
local last_refill = tonumber(bucket[2]) or now

-- Refill: add tokens based on elapsed time
local elapsed = math.max(0, now - last_refill)
tokens = math.min(capacity, tokens + elapsed * rate)

local allowed = 0
if tokens >= cost then
    tokens = tokens - cost
    allowed = 1
end

redis.call('hmset', key, 'tokens', tokens, 'last_refill', now)
redis.call('expire', key, math.ceil(capacity / rate * 2))

return {allowed, tokens}
"""

class TokenBucketLimiter:
    def __init__(self, redis_client, rate: float, capacity: int):
        self.r = redis_client
        self.rate = rate              # tokens per second
        self.capacity = capacity      # maximum in the bucket
        self.script = self.r.register_script(LUA_TOKEN_BUCKET)

    def consume(self, key: str, cost: int = 1) -> tuple[bool, int]:
        """
        Tries to consume `cost` tokens from `key`.
        Returns: (allowed, tokens_left)
        """
        now = time.time()
        result = self.script(keys=[key], args=[self.rate, self.capacity, now, cost])
        return bool(result[0]), int(result[1])

Usage in the API:

limiter_user_chat = TokenBucketLimiter(r, rate=1/60, capacity=60)  # 60/hr per user

@app.post("/api/chat")
def chat(req: ChatRequest, user_id: str):
    # Cost = 1 base + 1 per each 256 max_tokens
    cost = 1 + (req.max_tokens // 256)

    allowed, tokens_left = limiter_user_chat.consume(f"chat:{user_id}", cost=cost)
    if not allowed:
        raise HTTPException(
            429,
            detail={
                "error": "rate_limit_exceeded",
                "tokens_left": tokens_left,
                "retry_after_seconds": 60,
            }
        )

    # ... rest of the processing

---

Budget rate limiting

Converting requests → dollars. You follow the same pattern, but the "bucket tokens" are dollars (or LLM tokens).

# Budget bucket per tenant
# capacity = $100/month, rate = $100 / (30*24*3600) per second
budget_limiter = TokenBucketLimiter(
    r,
    rate=100 / (30 * 24 * 3600),  # refills $100 per month
    capacity=100,                  # max $100 accumulated
)

@app.post("/api/chat")
def chat(req: ChatRequest, tenant_id: str, user_id: str):
    # Estimate the request cost in cents × 100 = "millicents"
    # GPT-4o-mini: ~$0.15 input + $0.60 output / 1M tokens
    estimated_input = len(req.prompt) // 4  # rough tokens
    estimated_output = req.max_tokens
    cost_dollars = (
        estimated_input * 0.15 / 1_000_000
        + estimated_output * 0.60 / 1_000_000
    )
    cost_units = int(cost_dollars * 10000)  # decimals

    # Check the tenant's budget
    allowed, budget_left = budget_limiter.consume(f"budget:{tenant_id}", cost=cost_units)
    if not allowed:
        raise HTTPException(429, detail={
            "error": "budget_exceeded",
            "budget_left_cents": budget_left / 100,
        })

    # ... rest of the processing

What matters: once processed, you adjust the real cost (because real tokens may differ from the estimate):

# After the LLM call
actual_cost = response.usage.prompt_tokens * 0.15 / 1_000_000 + response.usage.completion_tokens * 0.60 / 1_000_000
diff_units = int((actual_cost - cost_dollars) * 10000)
if diff_units > 0:
    # Charge more than estimated
    budget_limiter.consume(f"budget:{tenant_id}", cost=diff_units)
elif diff_units < 0:
    # Return to the bucket (refund) — implement as an inverse consume
    pass

---

Behavior when the limit is reached

Three options, with clear trade-offs:

Hard reject (429)

The client receives 429 Too Many Requests. It has to retry later.

Pros: simple, clear. Cons: the client must handle the retry, bad UX if it happens often.

When to use it: public APIs, strict per-plan rate limits.

Queue (defer)

The client receives 202 + job_id. The worker eventually processes it when the bucket refills.

Pros: the client doesn't see an error, it can be transparent. Cons: variable latency (can take minutes during peak hours), the complexity of a per-user queue.

When to use it: non-interactive jobs (batch processing, async tasks).

Degrade

The client receives a degraded response — a cache hit, a response from a cheaper model, a truncated response.

Pros: UX maintained. Cons: requires implementing the degraded mode beforehand (see lesson 07).

When to use it: premium users with an SLA, cases where "something is better than nothing".

Typical combination: hard reject by default, queue for batch endpoints, degrade for paying-tier users.

---

Communicating rate limits to clients

Good APIs include headers that tell the client its state:

HTTP/1.1 200 OK
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 47
X-RateLimit-Reset: 1714512300

When you reach the limit:

HTTP/1.1 429 Too Many Requests
Retry-After: 30
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1714512300

{"error": "rate_limit_exceeded", "message": "Try again in 30 seconds"}

This lets smart clients back off without aggressive polling.

---

Coordinating with the LLM provider's rate limit

OpenAI limits you to 60 RPM and 90,000 TPM (tokens per minute). If your API accepts 200 RPM, the excess is going to fail at OpenAI. Your rate limit must respect the provider's.

Simple calculation: if the provider allows N RPM and you have K workers, each worker must limit itself to N/K RPM. Or use a shared rate limiter across workers (a centralized Redis token bucket).

# Shared across workers — global limit to OpenAI
openai_limiter = TokenBucketLimiter(r, rate=60/60, capacity=60)  # 60 RPM

def call_openai(prompt: str, max_tokens: int):
    # Wait your turn if necessary
    while True:
        allowed, _ = openai_limiter.consume("openai_global", cost=1)
        if allowed:
            break
        time.sleep(1)

    return openai_client.chat.completions.create(...)

This prevents your system from firing 100 requests/second at OpenAI and receiving 80 errors 429.

---

Common traps

Trap 1 — "Rate limit per IP." IP is a bad proxy for a user: users behind NAT share an IP, attackers rotate IPs, etc. Limit by authenticated user_id or API key, not by IP. If you have public endpoints, IP is an OK fallback but with lax limits.

Trap 2 — "Fixed cost per request." In AI, a request with 50 tokens of output costs 1/40 of one with 2000 tokens. If your rate limit uses a fixed cost, the first one "counts" the same as the second, which is unfair and badly calibrated. Proportional cost.

Trap 3 — "Rate limit only in my API." Your API accepts 200 RPM. The LLM provider limits to 60 RPM. The 140 excess RPM are going to fail. Design your rate limit aligned or stricter than the provider's.

Trap 4 — "429 without informative headers." The client receives a 429 and doesn't know when to retry. It resorts to aggressive retrying that worsens the problem. Always return Retry-After and the X-RateLimit-* headers.

Trap 5 — "A rate limit that's trivial to bypass." If your rate limit is implemented in the client-side JS, the attacker simply removes the code. Rate limit always on the server, ideally in a middleware before the business logic.

Trap 6 — "Not measuring how many requests I reject." Without a metric of "how many 429s I return per minute", you don't know if your rate limit is badly calibrated (too strict: legitimate clients suffer; too lax: it doesn't protect). Mandatory metric.

---

Exercise

Design the rate limiting strategy for this case:

System: an AI assistants SaaS. Free plan: 50 messages/day. Pro plan: 1000 messages/day. Enterprise plan: unlimited but with a cap of $5000/month on tokens. The backend uses GPT-4o-mini.

Provider constraints: OpenAI gave you tier 4: 10K RPM, 30M TPM.

Specify:

1. Algorithm? Why?
2. Rate limit levels (per user, tenant, global, vendor)
3. For enterprise, how do you convert "$5000/month" into operational rate limits?
4. Behavior when the limit is reached (hard reject, queue, degrade)
5. Headers you return to the client

See solution

1. Token bucket with variable cost. Allows reasonable bursts, lets you express "this request costs more" by max_tokens, easy to implement in Redis with Lua.
2. Levels:
   • Per free user: 50 req/day = bucket cap=50, rate = 50/86400 = ~0.000579 t/s
   • Per pro user: 1000 req/day = bucket cap=1000, rate = 1000/86400 = ~0.0116 t/s
   • Per enterprise tenant: budget of $5000/month on tokens, no rate limit by messages but yes by accumulated dollars (a budget bucket)
   • System-wide: 10K RPM (matching OpenAI's tier) = bucket cap=10000, rate=10000/60 ≈ 167 t/s
3. $5000/month on GPT-4o-mini with an avg of 700 tokens/request × $0.50/1M tokens ≈ $0.00035/req → ~14M possible requests. But the important operational cap is: rate = $5000 / (30243600) ≈ $0.0019/s. Convert to "cost units" by multiplying by 10000 (centi-cents) → rate = 19, bucket cap = $5000 × 10000 = 50M units. When a request costs more than tokens_left, reject.
4. Free / Pro: hard reject with 429 and a header indicating "your plan resets at midnight". Enterprise: degrade to GPT-4o-mini even if they asked for GPT-4o (fallback to a cheaper model when it approaches the cap).
5. Headers:
   • X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset (per user)
   • For enterprise: X-Budget-Remaining-Cents, X-Budget-Reset (when the month renews)
   • If 429: Retry-After: <seconds>

---

Summary

You learned:

• ✅ Algorithms: token bucket (default for AI), leaky bucket, fixed/sliding window
• ✅ Multi-level: per user, tenant, endpoint, global, vendor (all apply)
• ✅ Token bucket with variable cost (a request "costs" more based on max_tokens)
• ✅ Budget rate limiting: a bucket of dollars/cents, not just requests
• ✅ Behaviors: hard reject (429), queue, degrade
• ✅ Informative headers (X-RateLimit-*, Retry-After)
• ✅ Coordination with the provider's rate limits (don't accept more than the downstream tolerates)

Checkpoint: if you can design a rate limit that respects user, tenant, and vendor simultaneously, you're ready.

---

Next lesson

05 — Circuit breakers for LLM providers. Your queue processes well and your rate limiter protects. But what happens when OpenAI consistently takes 30s? Your workers get stuck, the queue grows, everything degrades. You're going to learn to detect silent degradation (not just errors) and to cut off the flow to a downed downstream before it drags down your whole system.

---

Resources

1. Stripe — Scaling rate limiters — classic write-up of how they do it.
2. Redis cell — token bucket as Redis module — ready-to-use implementation.
3. Cloudflare — How we built rate limiting — massive scalability.
4. OpenAI — Rate limits — understand your provider's.
5. Designing Data-Intensive Applications — chapter on throttling.