Module 1: Decision Framework for LLM Access

Common Mistakes When Choosing an LLM Provider

Capsule overview

Learning from other people's mistakes is cheaper than making them yourself.

This capsule documents the 7 most common mistakes when choosing an LLM provider, based on real experience from failed projects, costly pivots, and decisions that generated tech debt.

Each mistake includes:

🚫 What was done wrong
⚠️ Why it's problematic
✅ How to avoid it
📖 A real example

Goal: That you do NOT make these mistakes in your project.

❌ Mistake #1: Optimizing Cost Prematurely

What was done wrong:

"Ollama is free, I'm going to use it from day 1 to save on costs."

They choose local Ollama without evaluating other options, because $0/month operational is attractive. They ignore that setup takes 2 weeks, the team doesn't have the skills, and the hardware costs $3k.

Why it's problematic:

The REAL cost is higher:
- Hardware: $2-5k upfront
- DevOps time: 40-80 hours ($2-4k)
- Maintenance: $200-500/month
- Total year 1: $5-10k
Compared to OpenAI:
- Typical startup volume: 5k queries/day
- OpenAI cost: $75/month = $900/year
- Ollama is 5-10x MORE EXPENSIVE in year 1
Opportunity cost:
- 2 weeks of setup = no shipping features
- A competitor with OpenAI launches first

Real example:

Startup: E-commerce chatbot
Decision: Local Ollama to "save money"
Result:

Week 1-2: Ollama setup (server, GPU, networking)
Week 3: VRAM problems (the model won't load)
Week 4: Downgrade to a smaller model (worse quality)
Week 5: The CTO gives up, migrates to the OpenAI API in 1 day
Total cost: $4k wasted + 1 month lost

What they should have done:

Start with OpenAI ($75/month)
Validate product-market fit first
Migrate to Ollama LATER if it scales (>50k queries/day)

✅ How to avoid it:

Rule: Optimize cost AFTER validating product-market fit

MVP phase (0-6 months):
- Use the SIMPLEST option (OpenAI API)
- Focus on the product, not the infra
- Typical cost: $50-500/month (acceptable for validating)
Growth phase (6-18 months):
- If volume grows 10x, THEN evaluate alternatives
- OpenAI cost > $2k/month → Consider Ollama/OpenRouter
Scale phase (18+ months):
- High volume (millions of queries/month)
- An Ollama cluster makes sense (positive ROI)

Exception: On-premise MANDATORY (compliance) → Ollama from day 1 is valid.

❌ Mistake #2: Ignoring the Team's Skills

What was done wrong:

"Modal is the ideal architecture (serverless, autoscaling). Let's go with that."

They choose "best practice" technology without evaluating whether the team can implement it. A junior DevOps doesn't understand cold starts, doesn't know how to debug lambdas, and the project stalls.

Why it's problematic:

The learning curve eats the timeline:
- Junior learns Docker: 2 weeks
- Junior learns Modal: 1 week
- Junior debugs issues: 2 weeks
- Total: 5 weeks (vs 1 day with OpenAI)
Impossible maintenance:
- A junior can't resolve an outage alone
- The founders have to step in (not scalable)
Tech debt:
- Complex code nobody understands
- Fear of touching it (bugs in production)

Real example:

Startup: Content generation tool
Team: 2 founders (1 mid dev, 1 designer)
Decision: Modal serverless (they read a blog post)
Result:

Week 1-2: They followed the tutorial (it worked)
Week 3: Custom logic (doesn't work)
Week 4: Cold starts 10s (users complain)
Week 5: Stack Overflow, ChatGPT, don't solve it
Week 6: They hired a consultant ($3k) to fix it
Week 8: They migrated to the OpenAI API (they should have started that way)

Total cost: $3k consultant + 2 months lost

✅ How to avoid it:

Rule: Match the technology to real (not aspirational) skills

Skill Level	Can maintain	CANNOT maintain
Junior	OpenAI API, OpenRouter	Ollama cluster, Modal
Mid	Ollama single node, Modal	Ollama multi-node cluster
Senior	Everything above	N/A

The "bus factor" test:

If your main dev leaves, can the rest maintain the system?
If NOT → Choose simpler technology

Principle: Simplicity > Technical perfection

❌ Mistake #3: Not Having a Fallback Plan

What was done wrong:

"OpenAI is reliable, we don't need a backup."

They depend 100% on one provider with no contingency. The OpenAI API has a 4-hour outage, the application is completely down, and they lose revenue.

Why it's problematic:

Downtime = Revenue loss:
- E-commerce: $1k/hour lost (on average)
- B2B SaaS: Churning clients
- Reputation: an HN post "X is down again"
Every provider has outages:
- OpenAI: ~99.5% uptime (4 hours/month)
- Anthropic: Similar
- Local Ollama: Your responsibility
An emergency migration is chaotic:
- Without a plan: 6-12 hours to implement a fallback
- With a plan: 5 minutes (flip a switch)

Real example:

Startup: AI writing assistant
Revenue: $50k/month
Decision: OpenAI GPT-4 (no fallback)
Incident:

OpenAI outage: 6 hours (Saturday)
App completely down
200 users try to use it, it fails
Twitter: "X doesn't work, looking for an alternative"
Monday: 15 cancellations ($750 MRR lost)

Total cost: $750/month MRR + reputational damage

What they should have had:

OpenRouter configured as a fallback (5 min setup)
Auto-switch if OpenAI latency >5s for 2 min
Cost: $0 until the fallback activates

✅ How to avoid it:

Rule: Always have a Plan B ready (even if you never use it)

Basic fallback setup (30 minutes):

# 1. Abstract the provider behind an interface
class LLMProvider(Protocol):
    def chat(self, messages: list) -> str: ...

class OpenAIProvider(LLMProvider):
    def chat(self, messages): 
        return openai.chat.completions.create(...)

class OpenRouterProvider(LLMProvider):
    def chat(self, messages):
        return requests.post("https://openrouter.ai/api/v1/chat/completions", ...)

# 2. Configure primary + fallback
PRIMARY = OpenAIProvider()
FALLBACK = OpenRouterProvider()

# 3. Auto-fallback in production
def chat_with_fallback(messages):
    try:
        return PRIMARY.chat(messages)
    except (Timeout, APIError) as e:
        logger.error(f"Primary failed: {e}, using fallback")
        return FALLBACK.chat(messages)

Cost: $0 until you activate the fallback

❌ Mistake #4: Following the Hype Without Evaluating

What was done wrong:

"GPT-4 is the best model, I'm going to use it for everything."

They use GPT-4 ($30/1M tokens input) for simple queries where GPT-3.5 ($0.50/1M) works just the same. A bill 60x higher with no benefit.

Why it's problematic:

Overpaying for unnecessary quality:
- GPT-4: 86% MMLU
- GPT-3.5: 70% MMLU
- FAQ chatbot: 70% is enough
Slower speed:
- GPT-4: 3.2s latency
- GPT-3.5: 1.5s latency
- Users notice (bounce rate)
60x cost for no reason:
- 100k queries/month × 500 tokens = 50M tokens
- GPT-4: $1500/month
- GPT-3.5: $25/month
- Difference: $1475/month wasted

Real example:

Startup: Recipe recommendation
Decision: GPT-4 for "maximum quality"
Result:

Month 1: $50/month (low volume)
Month 3: $800/month (scale)
Month 6: $3200/month (CFO alert)
Audit: 90% of queries are simple ("suggest a recipe with chicken")
- GPT-3.5 gives the same answer
- They didn't need complex reasoning

Solution:

Classifier: Simple query → GPT-3.5, complex → GPT-4
90% of queries to GPT-3.5: $3200 → $400/month
Savings: $2800/month

✅ How to avoid it:

Rule: Use the cheapest model that meets the requirement

Methodology:

Define the minimum accuracy:
- FAQ chatbot: 70%
- Legal analysis: 85%
- Medical diagnosis: 95%
Test models in ascending order:
- Start: Mixtral 8x7B (70%) → Does it meet it?
- If NOT: GPT-3.5 (70%) → Does it meet it?
- If NOT: GPT-4 (86%) → It should meet it
Benchmark on YOUR data:
- Don't use general benchmarks (MMLU)
- Create a test set of 100 real queries
- Evaluate accuracy manually

Hybrid strategy (advanced):

Classifier: Simple query → cheap model
Complex query → expensive model
Optimized cost without sacrificing quality

❌ Mistake #5: Not Validating Against Official Documentation

What was done wrong:

"I saw this code on Stack Overflow, I'll copy it directly."

They use obsolete code, a deprecated API, or incorrect parameters. In production, it fails for no clear reason.

Why it's problematic:

APIs change fast:
- OpenAI SDK v0 → v1 (breaking changes 2023)
- Ollama API format changed 3 times in 2024
- Stack Overflow has 2020 code (obsolete)
Impossible debugging:
- Error message: "400 Bad Request"
- New docs: the model parameter is now mandatory
- You didn't know because you used old code
Security risks:
- Old code doesn't validate inputs
- Injection attacks possible

Real example:

Developer: Follows a YouTube tutorial (2022)
Code:

# Tutorial 2022 (OpenAI SDK v0)
import openai
openai.api_key = "sk-..."
response = openai.Completion.create(
    engine="text-davinci-003",  # DEPRECATED
    prompt="Hello",
    max_tokens=100
)

Result:

The code doesn't work (2024: SDK v1, new API)
3 hours of debugging
Reddit post: "The OpenAI API doesn't work, what do I do?"

Solution:

Read the official docs: https://platform.openai.com/docs
Correct code (2024):

from openai import OpenAI  # SDK v1
client = OpenAI(api_key="sk-...")
response = client.chat.completions.create(
    model="gpt-3.5-turbo",  # Chat API, not Completion
    messages=[{"role": "user", "content": "Hello"}]
)

✅ How to avoid it:

Rule: ALWAYS validate against the official docs (updated 2026)

Checklist:

Reliable sources (priority order):
- ✅ Official provider docs (always updated)
- ✅ Official GitHub (examples repo)
- ⚠️ Stack Overflow (check the date, votes)
- ⚠️ YouTube tutorials (check the date)
- ❌ Reddit comments (not peer-reviewed)
Red flags of obsolete code:
- openai.Completion.create() → Deprecated 2023
- engine="text-davinci-003" → Obsolete
- import openai without from openai import OpenAI → Old SDK
Up-to-date official docs:
- OpenAI: https://platform.openai.com/docs
- Ollama: https://ollama.com/docs
- OpenRouter: https://openrouter.ai/docs
- Modal: https://modal.com/docs
Test locally before production:
- Copy-pasted code? Test it first
- Works? Then deploy

❌ Mistake #6: Underestimating the Cost of Context

What was done wrong:

"I'm going to put the ENTIRE conversation history into every request."

Each message includes the last 50 messages (context), even though they only need 3-5. Cost explodes because input tokens are 10x more than necessary.

Why it's problematic:

Cost scales with context size:
- 3 messages: 300 tokens
- 50 messages: 5000 tokens (16.6x)
- Cost: 16.6x higher
Latency increases:
- More input tokens = more processing time
- 5000 tokens: +500ms vs 300 tokens
Context limit hit:
- GPT-3.5: 16k tokens max
- 50 messages × 100 tokens = 5k input
- Only 11k left for output
- Long conversations fail

Real example:

Startup: Customer support chatbot
Decision: Always include the last 50 messages
Result:

Month 1: $200/month (low volume)
Month 3: $2500/month (same volume!)
Audit: The average query uses 4800 input tokens
- 4500 tokens = unnecessary old history
- Only the last 5 messages are relevant

Solution:

Sliding window: last 5 messages
4800 → 600 input tokens
$2500 → $300/month
Savings: $2200/month (88%)

✅ How to avoid it:

Rule: Minimize context to what's NECESSARY

Strategies:

Sliding window (simplest):

# Only the last N messages
MAX_HISTORY = 5
conversation_history = messages[-MAX_HISTORY:]

Summarization (advanced):

# Every 10 messages, summarize the history
if len(messages) > 10:
    summary = summarize(messages[:-5])
    messages = [summary] + messages[-5:]

Relevance filtering (expert):

# Only messages relevant to the current query
relevant_messages = retrieve_relevant(query, messages)
context = relevant_messages[-5:]

Monitoring:

Log token usage per request
Alert if the average > 1000 tokens (review needed)

❌ Mistake #7: Not Measuring Real Performance

What was done wrong:

"Benchmarks say GPT-4 is better, we use it."

They blindly trust public benchmarks (MMLU) without testing on THEIR data. In production, GPT-3.5 works just the same for THEIR use case.

Why it's problematic:

General benchmarks ≠ your use case:
- MMLU: University exam (math, science)
- Your app: Recommending e-commerce products
- Different skills
Overpaying without validating:
- GPT-4: 60x more expensive
- Without measuring, you assume "it's better"
- In reality, GPT-3.5 is enough
Optimizations impossible:
- Without metrics, you don't know what to improve
- Latency? Accuracy? Cost?

Real example:

Startup: Email classifier (spam/not spam)
Decision: GPT-4 (86% MMLU, "it must be better")
Result:

Cost: $800/month
Accuracy: They didn't measure it (assumed it was perfect)
Audit: They created a test set of 1000 emails
- GPT-4: 94% accuracy
- GPT-3.5: 93% accuracy
- Mixtral: 91% accuracy
- Difference: 1-3% irrelevant for the business

Solution:

They switched to Mixtral (OpenRouter)
$800 → $80/month (10x cheaper)
91% accuracy is enough (3% error acceptable)

✅ How to avoid it:

Rule: Measure performance on YOUR data (not generic benchmarks)

Set up an evaluation pipeline:

Create a test set (100-1000 samples):

test_set = [
    {"input": "Hi, how are you?", "expected": "greeting"},
    {"input": "Track my order #123", "expected": "order_status"},
    # ... 98 more
]

Benchmark the models:

for model in ["gpt-3.5", "gpt-4", "mixtral"]:
    results = [evaluate(model, sample) for sample in test_set]
    accuracy = sum(r["correct"] for r in results) / len(results)
    latency = sum(r["latency"] for r in results) / len(results)
    cost = sum(r["tokens"] for r in results) * PRICE[model]
    
    print(f"{model}: {accuracy=}, {latency=}, {cost=}")

Choose the model that meets the minimum requirement:

If 85% accuracy is required and Mixtral gives 86% → Use Mixtral
GPT-4 with 94% is overkill (+9% doesn't justify 60x cost)

📊 Summary of Mistakes

Top 7 mistakes and how to avoid them:

Mistake	Impact	Fix
#1: Optimizing cost prematurely	$5-10k wasted + 1 month	Start simple, optimize LATER
#2: Ignoring the team's skills	Stalled project, tech debt	Match tech to real skills
#3: Not having a fallback plan	Downtime, revenue loss	Set up a Plan B (30 min)
#4: Following hype without evaluating	60x overpaying	Use the cheapest model that meets it
#5: Not validating against official docs	Broken code, 3hrs debugging	Official docs > Stack Overflow
#6: Underestimating the cost of context	16x unnecessary cost	Sliding window, only the last N
#7: Not measuring real performance	Paying for unnecessary accuracy	Test on YOUR data, not benchmarks

🎯 Checklist: Am I making any of these?

Before deciding on a provider, verify:

Am I optimizing cost before validating PMF? (Mistake #1)
Can my team MAINTAIN this solution? (Mistake #2)
Do I have a fallback plan ready? (Mistake #3)
Did I choose by hype or by evaluation? (Mistake #4)
Did I validate the code against the 2026 official docs? (Mistake #5)
Am I including only the necessary context? (Mistake #6)
Did I measure accuracy on MY data? (Mistake #7)

If any is ❌, STOP and fix it.

🔗 Additional resources

Postmortem Archive - Real outage cases
Cost Optimization Guide - Best practices
LLM Leaderboard - Up-to-date benchmarks
OpenAI Migration Guide (v0 → v1) - Avoid obsolete code

➡️ Next step

Next capsule: 08-mini-project-requirements-assessment.md

Now that you know what NOT to do, it's time for the integrative mini-project.

You'll apply the entire Module 1 framework (5 dimensions, options landscape, decision matrix, trade-offs) to a complete real case: E-commerce chatbot.

You'll make an informed decision and document your scorecard.

Reading time: 10-12 minutes
Next: 08-mini-project-requirements-assessment.md

← PreviousReal-World Use Cases: Justified Decisions Next →Mini-Project: Requirements Assessment - E-commerce Chatbot