Module 3: Scaling Fundamentals

Horizontal vs Vertical Scaling for AI

Capsule overview

The two fundamental strategies for scaling any system are horizontal scaling (more instances of the same service) and vertical scaling (more resources per instance). In traditional web systems, the short answer is "horizontal almost always" — more instances behind a load balancer scale beyond the limits of a single machine. But for AI systems, that simple answer fails in several cases. Sometimes vertical scaling is more cost-effective. Sometimes neither one helps because the bottleneck is in an external API outside your control.

In this capsule you learn when each strategy wins for AI specifically, with concrete examples. Horizontal scaling shines for concurrent throughput when the bottleneck is request processing, not LLM inference. Vertical scaling has genuine cases in AI: keeping a local model loaded in memory to avoid cold start, GPU memory for large batches, an embedding cache in memory. And you're going to learn the most important case: when no scaling helps because you saturate the LLM provider's rate limits — a situation that requires another strategy (queue + retry + multiple providers).

The goal isn't for you to memorize "horizontal when, vertical when". It's for you to understand the fundamentals enough to apply judgment in new cases. Once you understand what each strategy does and why, choosing between them is trivial.

Horizontal scaling: the reasonable default

What it does

Horizontal scaling = more instances of the same service behind a load balancer.

        Load Balancer
       /     |     \
   API #1  API #2  API #3
       \     |     /
        Shared State (DB, Cache)

Each instance is identical. The load balancer distributes requests. If one fails, the others keep going. If you need more capacity, you add more instances.

What it's for in AI

Horizontal scaling is the right strategy for:

1. Concurrent request handling: your API receives 100 simultaneous requests, each one waits 5s for the LLM. A single instance can hold ~50 concurrent connections. To support 200 concurrent users, you need 4 instances.

2. High availability: if an instance crashes, the others keep serving. Without this, downtime of 1 instance = total downtime.

3. Rolling deploys: deploy a new version on one instance at a time. The others keep serving. Without this, a deploy = full downtime.

4. Geographic distribution: instances in multiple regions for low latency for global users.

How to implement it

Typical stack:

Container orchestration: Kubernetes, ECS, Cloud Run
Load balancer: ALB, Nginx, Cloud Load Balancing
Auto-scaler: HPA in Kubernetes, AWS Auto Scaling, Cloud Run autoscaling
Externalized state: DB and cache shared between instances

Minimum configuration:

2+ instances minimum (HA)
Auto-scale between min and max according to a metric
Health checks to remove unhealthy instances
Graceful shutdown to drain connections before killing an instance

When horizontal scaling does NOT help

Here is the critical counterexample that doesn't appear in generic courses.

Case: your AI system makes calls to OpenAI gpt-4o. OpenAI gives you a rate limit of 60 requests/minute (RPM) in your tier. Your API receives 200 requests/minute.

What happens if you add 10 instances? Nothing good. Each instance makes requests to the LLM. The 200 requests/minute saturate the combined rate limit of all the instances. Adding 10 more = same rate limit, same 200 requests failing.

Lesson: horizontal scaling only helps if the bottleneck is inside your system. If it's in an external API, it's useless. You need another strategy (queue, multiple providers, batching).

Vertical scaling: when resources per instance matter

What it does

Vertical scaling = same instance, more resources (more CPU, more RAM, a more powerful GPU).

Before:          After:
┌──────────┐     ┌──────────┐
│ 4 vCPU   │     │ 16 vCPU  │
│ 16 GB    │  →  │ 64 GB    │
│ No GPU   │     │ A100 GPU │
└──────────┘     └──────────┘

A single instance, more capacity. Operationally simpler.

What it's for in AI

Genuine cases where vertical wins:

1. Local models in memory: if you self-host a model (Llama 3 70B, Mistral), you need enough RAM/VRAM to keep it loaded. 70B parameters in fp16 = ~140GB. You can't "horizontally scale" a 70B model with instances of 16GB each — you need a large instance.

2. Avoiding cold start: if the model takes 60s to load, you prefer an always-on instance with the model loaded over multiple instances that load/unload.

3. GPU for batch processing: the embedding worker processes batches of 100 docs per LLM call. A powerful GPU does this fast; multiple small GPUs are not equivalent.

4. Large in-memory caches: if you cache frequent embeddings in memory (not Redis), more RAM = more cache = a better hit rate.

5. Reducing coordination overhead: with 1 large instance, there are no distributed concerns. Operationally simpler for medium systems.

Concrete AI cases

GPU for inference:

1 H100 (80GB): can run Llama 3 70B at a reasonable speed
8 L4 (24GB each): you can't run 70B on any single one; you'd have to distribute the model (complex)
Cost: 1 H100 ~$3/hr, 8 L4 ~$8/hr — 1 large instance can be cheaper

Embedding worker:

1 instance with 4 A100 GPUs: processes 1M docs/day batch
4 instances with 1 A100 each: same capacity, more coordination overhead

When vertical scaling does NOT help

1. If the bottleneck is I/O wait on the LLM provider: having more CPU doesn't help if your instance is waiting 5s on each LLM call.

2. If you need HA: 1 instance = single point of failure. Vertical scaling alone doesn't provide HA.

3. If the model fits in a small instance and you need concurrent requests: horizontal is better.

4. If the provider has an instance size limit: cloud providers have maximum instance types. If you need more, horizontal is a must.

Combining: horizontal AND vertical

In practice, most AI systems use both strategies in different components:

API Layer (horizontal):
  4 medium instances, auto-scaling according to concurrent requests

Embedding Worker (vertical):
  1 large instance with a dedicated GPU, batch processing

Vector DB (horizontal with replicas):
  3 nodes for HA + read scaling

Redis Cache (vertical):
  1 large instance with lots of RAM

Each component with the strategy its workload needs. There's no universal rule "always horizontal" or "always vertical".

The special case: the LLM provider's rate limit

This deserves its own section because it's where the most teams go wrong.

The problem

OpenAI gives you a tier with a rate limit (examples Q1 2026):

Free tier: 3 RPM, 200 RPD
Tier 1: 500 RPM, 10k RPD
Tier 2: 5,000 RPM, 30k RPD
Tier 3: 5,000 RPM, 100k RPD
Tier 4 (max): 10,000 RPM

If your system needs 500 RPM and you're on Tier 1 (also 500 RPM), you're right at the edge. Spikes are going to make you fail.

Anthropic is similar. Each provider has tiers that depend on your billing history.

Why scaling doesn't solve this

If you have 5 instances and each one makes requests, they all share the rate limit. 100 RPM per instance × 5 instances = 500 RPM combined, exactly the limit. You didn't add capacity — you distributed the same.

Adding instance 6 doesn't make OpenAI give you more RPM. It only divides the bottleneck into more pieces.

Real strategies

1. Upgrade your tier: you pay more, more rate limit. An obvious but limited solution (tiers are discrete, not infinite).

2. Queue + Worker pool: instead of making synchronous LLM calls from each API instance, you enqueue and a fixed-size worker pool processes at the provider's rate. If you saturate, requests wait in the queue (acceptable if async pattern).

3. Multiple providers + fallback: use OpenAI primary, Anthropic secondary. When OpenAI rate-limits, fallback to Anthropic. This effectively doubles your capacity.

4. Aggressive caching: if 50% of queries are repeated, a semantic cache reduces LLM calls by 50%. This effectively doubles your capacity without paying more to the provider.

5. Tier hierarchy: simple queries → a cheap model (more rate limit), complex queries → a premium model. This saves the premium model's rate limit.

6. Batch API: OpenAI offers a Batch API with a 50% discount and separate rate limits for non-time-sensitive processes.

Architectural implication

If you expect to hit the provider's rate limit, your architecture must have:

A queue layer between the API and the LLM calls
Retry with exponential backoff
A circuit breaker pattern (when you saturate, fallback)
Metrics of rate limit consumption (you can see it coming)

This is what's discussed in M5 (Reliability at Scale). But the seed is here: scaling AI requires accommodating external rate limits.

Worked example: applying both strategies

System: an internal Q&A API scaled to 50k queries/day (the internal Q&A already known from the previous module).

Analysis

Components:

API Layer: receives Slack events, orchestrates the flow
RAG Retrieval: vector search in Pinecone
LLM Client: calls to Anthropic
Embedding Worker: processes documents in batch
Cache: Redis

50k queries/day = ~35 RPM average (24h); with a non-uniform distribution, the peak is ~70 RPM (2x the average)

Scaling decisions per component

API Layer — horizontal:

3 baseline instances (HA)
Auto-scale 3-10 according to concurrent connections
~20 connections per instance × 5 = 100 concurrent supported
Stateless (state in Redis/Postgres)

RAG Retrieval — N/A:

Pinecone is managed, they handle its scaling
Your API only makes queries, you don't need to scale the vector search side

LLM Client — depends on the rate limit:

Anthropic Tier 3: 4,000 RPM. Your peak is ~70 RPM. Not a bottleneck.
If it were Tier 1 (500 RPM): 70 RPM peak is fine, but a small margin of error
No action needed currently; monitor RPM consumption

Embedding Worker — vertical:

1 large instance with a dedicated GPU
Processes batches of 1k docs/hour (when there are new uploads)
Sleeps when there's no queue
Vertical because batch processing benefits from a single powerful GPU

Cache (Redis) — vertical initially:

1 instance with 16GB RAM (enough for the hot cache)
Backup snapshots for basic HA
If it grows to >32GB, consider a Redis cluster (horizontal)

Auto-scaling configuration

# Kubernetes HPA for the API
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: qa-api
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: External
    external:
      metric:
        name: redis_queue_depth
      target:
        type: AverageValue
        averageValue: "10"  # Scale up if queue >10 messages per replica
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 70

Notice: the primary scaling trigger is queue depth (AI-specific), not CPU.

Common pitfalls and mistakes

Mistake 1: "Horizontal always"

An engineer applies horizontal scaling reflexively without evaluating the bottleneck. It works for some components, fails for others.

How to detect it: do you add instances when you hit the LLM provider's rate limit? Or when the bottleneck is a managed vector DB that already scales internally?

How to fix it: identify the bottleneck first (capsule 03). If it's in an external component or a single resource, horizontal doesn't help.

Mistake 2: Vertical scaling to avoid setting up horizontal

An engineer prefers "a bigger instance" because it's simpler than configuring Kubernetes and a load balancer. Result: a single point of failure, no HA, downtime on deploys or crashes.

How to detect it: do you run production on 1 instance "because it's simpler"? Your HA is 0%.

How to fix it: a minimum of 2 instances with a load balancer. The setup overhead of horizontal is worth it for basic HA.

Mistake 3: Auto-scaling on CPU for I/O bound services

CPU at 5% while 100 connections wait for the LLM. The auto-scaler says "it's fine", but the reality is it's saturated. Users see slowness or errors.

How to detect it: does your auto-scaler have CPU as the primary metric? Have you seen cases where alerts didn't fire but users reported problems?

How to fix it: auto-scale on queue depth, concurrent connections, or response time p99 — metrics that reflect the reality of I/O bound services.

Mistake 4: Not considering rate limits in planning

An engineer plans for 1M requests/day without checking OpenAI's tier. In production they discover they're on Tier 1 (500 RPM = 720k requests/day max). The product is DOA.

How to detect it: have you checked the provider's rate limit for your target volume?

How to fix it: check rate limits before design. If your volume exceeds the current tier, plan an upgrade or a multi-provider strategy.

Self-check

For an AI system you know, identify the appropriate scaling strategy per component:

Component	Strategy	Reason	Approximate configuration
API Layer	?	?	?
LLM Calls	?	?	?
Vector DB	?	?	?
Cache	?	?	?
Workers (background)	?	?	?

See example case (Q&A system 50k/day)

Component	Strategy	Reason	Configuration
API Layer	Horizontal	Stateless, concurrent requests	3-10 instances, HPA on queue depth
LLM Calls	N/A (rate limit)	External bottleneck	Anthropic Tier 3, monitor RPM
Vector DB	N/A (managed)	Pinecone handles scaling	Appropriate index size, ~$560/month
Cache	Vertical	Single point of hot data	1 Redis instance 16GB
Embedding Worker	Vertical	GPU batch processing	1 instance with A100

Summary and next step

In this capsule you saw the two fundamental strategies applied to AI:

Horizontal scaling: the default for concurrent throughput, HA, rolling deploys. Fails when the bottleneck is external (rate limits)
Vertical scaling: genuine cases in AI — large local models, GPU, in-memory caches, operational simplicity
Combinations: each component with its appropriate strategy
The LLM provider's rate limits: a special case where scaling doesn't solve it, requires queue/cache/multiple providers

Before moving on to capsule 03, you should be able to:

Distinguish horizontal from vertical with concrete criteria
Identify 3+ AI cases where vertical scaling is appropriate
Explain why horizontal scaling doesn't solve the LLM provider's rate limits
Decide strategy per component, not for the whole system

In capsule 03 — Bottlenecks in AI systems — you're going to dive into where the real bottlenecks are in AI systems. You're going to see it with numbers: how many ms each component spends, what percentage of the flow, where optimizing has an impact and where it doesn't. And you're going to learn bottleneck analysis — a method to identify the limiting component with math, not intuition. Without this skill, scaling is blind: you add resources where they don't help.

Resources

Scalability Lessons Learned — High Scalability Blog — Real cases of scaling in distributed systems
Horizontal vs Vertical Scaling — AWS Documentation — Official guides
Kubernetes Autoscaling Best Practices — Official docs
OpenAI Rate Limits — Official documentation on rate limits and tiers
Anthropic Rate Limits — Anthropic's rate limits
GPU Pricing Comparison — Comparison of GPU pricing between providers

← PreviousIntroduction to Scaling Fundamentals Next →Bottlenecks in AI Systems