Module 8: Your First AI System Design
4. Design Trade-Offs: Cost vs Latency vs Quality
Description
Reality: There's no "perfect" design. There are always trade-offs.
3 main dimensions:
- Cost: How much $ per request?
- Latency: How long does it take to respond?
- Quality: How good is the response?
The "trilemma": You can't maximize all 3 simultaneously.
- Maximum quality (GPT-4) → high cost, high latency.
- Minimum cost (GPT-3.5) → lower quality.
- Minimum latency (cache) → static responses (lower personalized quality).
In this lesson you'll understand common trade-offs and how to make decisions.
Dimension 1: Cost
What Impacts Cost
Factors:
- The model: GPT-4 ($0.03/1K input) vs GPT-3.5 ($0.0005/1K input) → a 60× difference.
- Volume: 1K requests/month vs 1M requests/month.
- Prompt length: Long prompts (1K tokens) vs short ones (100 tokens).
- Response length: max_tokens=100 vs max_tokens=1000.
- The pattern: A simple chatbot (1 call) vs RAG (embeddings + an LLM call) vs Agents (multiple calls).
Trade-Off: GPT-4 vs GPT-3.5
Example: An FAQ chatbot
| Aspect | GPT-4 | GPT-3.5 |
|---|---|---|
| Quality | Very high (superior reasoning) | High (enough for an FAQ) |
| Cost/request | $0.006 (200 input, 200 output) | $0.0002 (30× cheaper) |
| Cost/month (10K requests) | $60 | $2 |
Decision:
- If it's a simple FAQ (direct answers) → GPT-3.5 (that's enough).
- If it's a complex FAQ (reasoning) → GPT-4.
Trade-Off: Cloud API vs Self-Hosted
Example: 1M requests/month
| Aspect | Cloud (the GPT-4 API) | Self-Hosted (Llama 3 70B) |
|---|---|---|
| Cost | $6,000/month (variable) | $1,500/month for the GPU (fixed) |
| Quality | Very high (GPT-4) | High (Llama 3 70B ~80% of GPT-4) |
| Setup | 5 minutes | 1-3 days |
Decision:
- If volume is low (<100K requests/month) → a cloud API (cheaper).
- If volume is high (>1M requests/month) → self-hosted (4× cheaper).
Dimension 2: Latency
What Impacts Latency
Factors:
- The LLM: An API call (200-500ms) vs self-hosted locally (50-200ms).
- The pattern: A simple chatbot (1 call, <1s) vs RAG (embed + search + the LLM, 1-3s) vs Agents (multiple calls, 3-10s).
- max_tokens: 100 tokens (fast) vs 1000 tokens (slow).
- Cache: A hit (instant) vs a miss (an API call).
- The network: A US-based API call from LATAM (300ms) vs from the US (100ms).
Trade-Off: Quality vs Latency
Example: A RAG System
| Aspect | RAG with GPT-4 | A simple chatbot with GPT-3.5 |
|---|---|---|
| Latency | 2-3s (embed + search + the LLM) | <1s (only the LLM) |
| Quality | Very high (information specific to the docs) | Medium (general information) |
Decision:
- If the user tolerates 2-3s and needs specific information → RAG.
- If the user needs an instant response (<1s) and it's a general FAQ → a simple chatbot.
Trade-Off: GPT-4 vs Cache
Example: A repetitive FAQ
| Aspect | GPT-4 (every request) | Cache (a pre-generated response) |
|---|---|---|
| Latency | 500ms (an API call) | <50ms (a Redis lookup) |
| Quality | Personalized (the user's context) | Static (the same response every time) |
| Cost | $0.006/request | $0 (free after the first generation) |
Decision:
- If it's a frequent question ("What is AI?") and the answer doesn't change → cache.
- If it's a personalized question ("How can I improve my code?") → GPT-4.
Dimension 3: Quality
What Impacts Quality
Factors:
- The model: GPT-4 (better reasoning) vs GPT-3.5 (enough for simple tasks).
- Prompt engineering: A clear system prompt vs a vague one.
- RAG: With docs (specific information) vs without docs (general information).
- Fine-tuning: A model adapted to a specific case vs a base model.
- Temperature: Low (deterministic) vs high (creative but inconsistent).
Trade-Off: Quality vs Cost
Example: A Ticket Classifier
| Aspect | GPT-4 | GPT-3.5 |
|---|---|---|
| Accuracy | 95% | 90% |
| Cost/request | $0.002 | $0.00005 (40× cheaper) |
| Cost/month (100K tickets) | $200 | $5 |
Decision:
- If accuracy is critical (e.g. medical classification) → GPT-4.
- If 90% accuracy is acceptable (e.g. support tickets) → GPT-3.5 (40× cheaper).
Trade-Off: Quality vs Latency (Agents)
Example: An Agent vs RAG
| Aspect | An Agent (multi-tool) | RAG (a single retrieval) |
|---|---|---|
| Quality | Very high (multi-step reasoning) | High (single-step retrieval) |
| Latency | 5-10s (multiple calls) | 2-3s (1 call) |
| Cost | High (multiple LLM calls) | Medium (1 LLM call) |
Decision:
- If the task is complex (multi-step reasoning) → an agent (the user tolerates 10s).
- If the task is simple (Q&A) → RAG (2-3s is enough).
The Trilemma: You Can't Have Everything
Visualization:
High Quality
/\
/ \
/ \
/ \
/ \
Low Cost ---- Low Latency
Reality:
- High quality + low latency → high cost (GPT-4, no cache).
- High quality + low cost → high latency (self-hosted Llama 3, not optimized).
- Low latency + low cost → lower quality (a cache of static responses).
Optimization Strategies
1. A Multi-Model Strategy
Problem: Using GPT-4 for everything (expensive).
Solution:
- GPT-3.5 for simple tasks (80% of requests) → $2/10K.
- GPT-4 for complex tasks (20% of requests) → $12/10K.
- Total: $14/10K (vs $60/10K with pure GPT-4 → a 75% saving).
2. Intelligent Caching
Problem: Repetitive FAQs burn tokens.
Solution:
- The first time: GPT-4 generates the response → it's saved in the cache (Redis).
- The second time (the same question): It returns from the cache (instant, free).
- Saving: 90% of the cost on frequent questions.
3. Prompt Optimization
Problem: Long prompts (1K tokens) cost $.
Solution:
- A concise prompt (200 tokens) → an 80% saving.
- Few-shot examples (only 2-3, not 10).
4. Batch Processing
Problem: 1M individual requests (high overhead).
Solution:
- Batch 100 requests into 1 call (e.g. for classification).
- Saving: 50% of the cost (less overhead).
Real Use Cases
Case 1: A Startup MVP (a Limited Budget)
Requirements:
- An FAQ chatbot.
- Budget: $50/month.
- Volume: 5K requests/month.
Decision:
- Model: GPT-3.5-turbo (enough for an FAQ).
- Cache: Redis for frequent questions (70% of requests).
- Cost: $10/month (OpenAI) + $10/month (hosting) = $20/month.
Accepted trade-off: Quality slightly below GPT-4 (but 20× cheaper).
Case 2: A Company (High Volume, Critical Quality)
Requirements:
- Q&A over technical documentation.
- Volume: 500K requests/month.
- Critical accuracy (>95%).
Decision:
- Pattern: RAG (external docs).
- Model: GPT-4 (accuracy >95%).
- Vector DB: Pinecone (cloud, scalable).
- Cache: Redis for frequent queries (30% of requests).
- Cost: $3K/month (OpenAI) + $200/month (Pinecone) + $100/month (hosting) = $3.3K/month.
Accepted trade-off: A high cost (but accuracy is critical for the business).
Case 3: A Startup with Sensitive Data
Requirements:
- An internal chatbot (confidential data).
- Volume: 50K requests/month.
- HIPAA compliance (data can't leave the infrastructure).
Decision:
- Self-hosted: Llama 3 (70B) on your own servers.
- Vector DB: Chroma (local).
- Cost: $1,500/month (the GPU) + $200/month (hosting) = $1.7K/month.
Accepted trade-off: A high fixed cost (but guaranteed privacy).
Why this matters for an AI Engineer
1. Informed decisions
The Product Manager: "Why don't we use GPT-4 for everything?"
An AI Engineer (without understanding the trade-offs): "Okay, we'll use GPT-4."
An AI Engineer (understanding the trade-offs): "GPT-4 costs $60/10K requests. GPT-3.5 costs $2/10K. For a simple FAQ, GPT-3.5 is enough. Saving: 30×. I recommend a multi-model strategy."
2. Continuous optimization
Monitoring: "80% of the cost goes to GPT-4 on simple questions."
Action: Migrate the simple questions to GPT-3.5 → a 70% saving.
Summary
3 dimensions:
- Cost: GPT-4 (expensive) vs GPT-3.5 (30× cheaper).
- Latency: A simple chatbot (<1s) vs RAG (2-3s) vs Agents (5-10s).
- Quality: GPT-4 (high) vs GPT-3.5 (enough for simple tasks).
The trilemma: You can't maximize all 3 simultaneously.
Optimization strategies:
- A multi-model strategy (GPT-3.5 for simple, GPT-4 for complex).
- Intelligent caching (frequent FAQs → Redis).
- Prompt optimization (concise prompts).
- Batch processing (multiple requests in 1 call).
Decisions based on:
- Budget (how much can you spend?).
- Volume (1K or 1M requests/month?).
- Required accuracy (90% or 95%?).
- Tolerated latency (<1s or is 5s fine?).
Next step: Lesson 05: A Case Study — A complete design of a Q&A system (step by step).