Module 1: Decision Framework for LLM Access
Real-World Use Cases: Justified Decisions
Capsule overview
The best way to learn the framework is by seeing real cases where others applied it.
This capsule presents 5 real (anonymized) projects with full context, an evaluation scorecard, the decision made, and a retrospective 6 months later. You'll see:
- ✅ Decisions that worked (and why)
- ⚠️ Accepted trade-offs
- ❌ Mistakes (so you don't repeat them)
All the cases come from real startup, company, and side-project work between 2024-2026.
📖 Case 1: LegalBot - Privacy Priority
Context:
Company: Mid-size law firm (50 lawyers)
Project: Chatbot for searching past cases (case law)
Volume: 200 queries/day (internal lawyers)
Data: CRITICAL (client case files, absolute confidentiality)
Stakeholders:
- Founding partner: "Data CANNOT leave the server, it's a clause in the client contract"
- IT Manager: A team of 2 people (1 junior DevOps, 1 helpdesk)
- Budget: $10k setup, $1k/month operational
Scorecard:
| Dimension | Priority | Evaluation |
|---|---|---|
| Privacy | CRITICAL | On-premise MANDATORY |
| Cost | Important | Limited but existing budget |
| Quality | Important | Case-law search (95%+ accuracy not critical) |
| Speed | Secondary | Internal users (5-10s acceptable) |
| Simplicity | Important | Junior team (needs mentoring) |
Decision: Local Ollama (Mixtral 8x7B)
Justification:
Meets privacy (CRITICAL dimension):
- 100% on-premise (local server in the office)
- Data NEVER leaves the internal network
- Compliance with confidentiality clauses
Cost within budget:
- Hardware: $5k server with RTX A6000 (professional GPU)
- Setup: DevOps consultant 2 weeks = $3k
- Operational: $0/month (only electricity ~$50/month)
- Total year 1: $8600 < $22k budget (setup $10k + operational $1k × 12)
Sufficient quality:
- Mixtral 8x7B: 70.6% MMLU (enough for search)
- RAG over the case-law database (not purely generative answers)
Accepted trade-offs:
- Speed: 8s/query (vs 2s with OpenAI) → OK because users are internal
- Simplicity: Setup 2 weeks with a consultant → OK because privacy is NON-negotiable
Retrospective (6 months later):
✅ Worked:
- 100% compliance (audit passed without issues)
- Final cost: $50/month electricity (as expected)
- Satisfied users (8s speed acceptable)
⚠️ Challenges:
- First week: Problems with VRAM (Mixtral 8x7B requires 48GB)
- Solution: Downgrade to Mixtral 8x7B quantized (4-bit)
- Month 3: The model gets corrupted after an Ollama update
- Solution: Model versioning + backups
❌ If they did it again:
- Hire a senior DevOps (not junior) from the start
- Budget 20% more for hardware (GPUs go up in price)
Rating: 8/10 ✅ Correct decision
📖 Case 2: TravelAssist - Startup MVP
Context:
Company: Pre-seed startup (3 founders)
Project: Chatbot for travel recommendations (MVP)
Volume: 500 beta users, ~2k queries/day
Timeline: 2 weeks for the investor demo
Stakeholders:
- CEO: "I need a working demo NOW, investors in 2 weeks"
- CTO (technical founder): Mid-level Python, no DevOps
- Budget: $300/month (founders bootstrapping)
Scorecard:
| Dimension | Priority | Evaluation |
|---|---|---|
| Simplicity | CRITICAL | 2-week timeline, small team |
| Speed | CRITICAL | The demo UX must impress |
| Cost | Important | Tight but existing budget |
| Quality | Important | Recommendations must be coherent |
| Privacy | Secondary | Public data (tourist destinations) |
Decision: OpenAI API (GPT-3.5-turbo)
Justification:
Meets simplicity (CRITICAL):
- Setup: <4 hours (OpenAI SDK + FastAPI + frontend)
- Deploy: Vercel frontend + Railway backend = 1 day total
- Zero infrastructure (the CTO can focus on the product)
Meets speed (CRITICAL):
- 1.5s latency (impresses in the demo)
- Streaming responses (smooth UX)
Cost within budget:
- 2k queries/day × 30 days = 60k queries/month
- 60k × 500 tokens × $0.002/1k = $60/month
- ✅ Well within the $300 budget
Accepted trade-offs:
- Privacy: Cloud (OK because data is public)
- Vendor lock-in: OpenAI-specific (OK for an MVP, refactor later if it scales)
Retrospective (6 months later):
✅ Worked:
- Successful demo: They raised a $500k seed round
- Growth: 500 → 5k users (10x)
- Cost scaled well: $60 → $600/month (linearly)
⚠️ Challenges:
- Month 4: OpenAI API down 2 hours (they lost users)
- Solution: They implemented OpenRouter as a fallback (Module 5)
- Month 5: A $900 bill (OpenAI changed pricing)
- Solution: They switched simple queries to Mixtral (OpenRouter)
🔄 Next step:
- Month 7: Evaluating a partial migration to Ollama for simple queries
- Keep OpenAI for complex queries (hybrid strategy)
Rating: 9/10 ✅ Excellent decision for an MVP
📖 Case 3: HealthTech - Enterprise with Compliance
Context:
Company: Enterprise healthcare company (1500 employees)
Project: Assistant for reviewing medical notes (sentiment analysis)
Volume: 50k notes/day (multi-hospital)
Data: HIPAA-regulated (sensitive medical information)
Stakeholders:
- Compliance Officer: "HIPAA is NON-negotiable, fines of $100k+"
- CTO: A DevOps team of 10 senior people
- CFO: Budget $200k/year (including hardware)
Scorecard:
| Dimension | Priority | Evaluation |
|---|---|---|
| Privacy | CRITICAL | HIPAA mandatory (on-premise) |
| Scalability | CRITICAL | 50k queries/day (multi-hospital) |
| Cost | Important | High but finite budget |
| Quality | Important | Sentiment analysis (not diagnosis) |
| Speed | Secondary | Background processing (async OK) |
Decision: Ollama cluster (8 nodes with A100 GPUs)
Justification:
Meets privacy (CRITICAL):
- 100% on-premise (hospital datacenter)
- Zero data egress (compliance guaranteed)
- BAA not needed (no third parties involved)
Meets scalability (CRITICAL):
- 8 nodes × 6k queries/day each = 48k queries/day capacity
- Horizontal scaling (add nodes if it grows)
- Load balancer (distribute queries uniformly)
Justified cost:
- Hardware: $120k (8 servers × $15k each)
- Setup: $30k (consultant + internal team 2 months)
- Operational: $2k/month (electricity + maintenance)
- Total year 1: $174k < $200k budget
Sufficient quality:
- Llama 2 70B: 68.9% MMLU (enough for sentiment analysis)
- Fine-tuning with historical data (improves accuracy +10%)
Accepted trade-offs:
- Simplicity: Complex setup (2 months) → OK because it's a senior team
- Speed: 4s/query (vs 1.5s OpenAI) → OK because it's a background job (not real-time)
Retrospective (12 months later):
✅ Worked:
- HIPAA audit: Passed without issues (zero findings)
- Scalability: They grew to 80k queries/day (added 4 nodes)
- Year 2 cost: $60k hardware + $24k operational = $84k/year
- vs hypothetical OpenAI: 80k × 30 × 500 tokens × $0.002/1k = $2.4M/year
- Savings: $2.3M/year ✅
⚠️ Challenges:
- Month 2: GPU failure (1 node down 48hrs)
- Solution: Spare GPU (they now have 9 nodes, 1 spare)
- Month 6: Model drift (accuracy dropped 5%)
- Solution: Quarterly re-training with new data
❌ If they did it again:
- Include 2 spare nodes from the start (redundancy)
- Hire an ML engineer for fine-tuning (the current team didn't have the expertise)
Rating: 10/10 ✅ The only viable option (HIPAA)
📖 Case 4: EventBot - Variable Traffic
Context:
Company: Series A startup ($5M funding)
Project: Chatbot for events (concerts, sports)
Volume: Variable (1k queries/day normal, 50k during big events)
Spikes: Super Bowl, World Cup final = 200k queries/day
Stakeholders:
- CEO: "We can't lose users during big events (critical revenue)"
- CTO: A team of 5 mid-senior Python devs
- CFO: $5k/month fixed budget (no cost spikes)
Scorecard:
| Dimension | Priority | Evaluation |
|---|---|---|
| Scalability | CRITICAL | 1k → 200k queries/day (200x spike) |
| Predictable cost | CRITICAL | The CFO demands $5k/month flat |
| Speed | Important | Live-event UX (real-time) |
| Simplicity | Important | A mid-level team can maintain it |
| Privacy | Secondary | Public data (events) |
Initial decision: OpenAI API
Problem discovered in month 2:
- Super Bowl Sunday: 180k queries/day
- OpenAI bill: $5400 THAT DAY ($162k/month if it were constant)
- CFO: "The budget is $5k/month, this is unacceptable"
Pivot to: Modal serverless
Justification for the pivot:
Meets scalability (CRITICAL):
- Autoscaling: 0 → 500 instances in 2 minutes
- Cold starts later optimized (warm pool)
More predictable cost:
- Normal days: $50/day = $1500/month ✅
- Super Bowl: $2000 THAT DAY (vs $5400 OpenAI)
- Monthly average: $2500/month (2 spikes/month) ✅ < $5k budget
Accepted trade-offs:
- Setup complexity: 1 week (vs 1 day OpenAI) → OK because it's a mid-senior team
- Cold starts: First request 3s (vs 1.5s) → Warm pool solves it
Retrospective (8 months later):
✅ Worked:
- 6 big events handled with no downtime
- Average cost: $2800/month (within budget)
- Happy CFO (predictable)
⚠️ Challenges:
- Month 1 post-migration: Cold starts problem (3-5s on the first request)
- Solution: Warm pool (keep 10 instances always on)
- Month 4: Modal changed pricing (+20%)
- Solution: They renegotiated with Modal (enterprise plan, flat rate)
🔄 Evaluating:
- Month 8: Considering an Ollama cluster for baseline traffic (1k queries/day)
- Modal only for the spikes (hybrid)
- Potential savings: $1000/month
Rating: 7/10 ⚠️ Works but cost is still a challenge
📖 Case 5: CodeReviewer - Side Project
Context:
Type: Open-source side project (1 maintainer)
Project: A CLI tool for AI code reviews
Volume: 1k users, ~5k queries/month (low)
Budget: $0 (personal project)
Constraints:
- Maintainer: Full-time job, 5 hrs/week available
- Budget: Zero (no monetization)
- Data: Public GitHub repos (not sensitive)
Scorecard:
| Dimension | Priority | Evaluation |
|---|---|---|
| Cost | CRITICAL | $0 budget (no exceptions) |
| Simplicity | CRITICAL | 5 hrs/week available |
| Privacy | Secondary | Public repos |
| Quality | Important | The code review must be useful |
| Speed | Secondary | CLI tool (users wait) |
Decision: LM Studio (development) + Ollama (production)
Justification:
Meets cost (CRITICAL):
- LM Studio: Free (maintainer's M2 Mac laptop)
- Ollama: Free (users self-host)
- Operational: $0/month
Meets simplicity (CRITICAL):
- LM Studio setup: 15 minutes
- Docs for users: "Install Ollama + Pull Mistral 7B"
- Maintenance: 1 hr/month (only bug fixes)
Strategy:
- The maintainer uses LM Studio locally (development)
- Users use Ollama (each on their own machine)
- The CLI tool calls
localhost:11434(Ollama default)
Accepted trade-offs:
- Quality: Mistral 7B (62.5% MMLU) vs GPT-4 (86%) → Enough for code review
- Speed: 10-15s/review (vs 2s OpenAI) → CLI users accept it
- UX: Users must install Ollama (friction) → Clear documentation mitigates it
Retrospective (18 months later):
✅ Worked:
- 1k → 10k users (10x growth)
- Cost: $0 (stayed free)
- The community contributed installation docs (reduced support)
⚠️ Challenges:
- 30% of users can't install Ollama (Windows issues)
- Solution: They added a "bring your own OpenAI key" option
- 15% complain about speed (10s)
- Solution: Docs explain the trade-off (free vs fast)
🔄 Next step:
- Consider a freemium model:
- Free tier: Local Ollama (current)
- Paid tier ($5/month): OpenAI API (hosted)
- It could generate $500/month (100 users × $5) for sustainability
Rating: 9/10 ✅ The only viable option with a $0 budget
📊 Case Summary
Comparison table:
| Case | CRITICAL dimension | Decision | Year-1 cost | Rating | Lesson |
|---|---|---|---|---|---|
| LegalBot | Privacy | Local Ollama | $8.6k | 8/10 | On-premise only if NON-negotiable |
| TravelAssist | Simplicity + Speed | OpenAI API | $720 | 9/10 | OpenAI perfect for MVPs |
| HealthTech | Privacy + Scalability | Ollama cluster | $174k | 10/10 | Compliance justifies any cost |
| EventBot | Scalability + Cost | Modal | $33.6k | 7/10 | Serverless only if traffic is variable |
| CodeReviewer | Cost ($0) | LM Studio/Ollama | $0 | 9/10 | Local is the only option with a $0 budget |
🔗 Additional resources
- Case Studies Collection - More real cases
- Ollama Deployment Guide - Production setup
- OpenAI Case Studies - Enterprise examples
- r/LocalLLaMA - Community local cases
➡️ Next step
Next capsule: 07-common-mistakes-when-choosing.md
Now that you've seen successful decisions, you'll see the most common mistakes when choosing a provider:
- Optimizing cost prematurely
- Ignoring the team's skills
- Not having a fallback plan
- Following hype without evaluating
Learn what NOT to do.
Reading time: 12-15 minutes
Next: 07-common-mistakes-when-choosing.md