Module 1: Decision Framework for LLM Access

The Decision Matrix: A Structured Framework for Choosing an LLM Provider

Capsule overview

Now that you know the 5 evaluation dimensions (capsule 02) and the landscape of available options (capsule 03), you need a structured framework to make the final decision.

This capsule gives you that framework: a decision matrix that maps specific requirements to recommended providers. It's not "opinion" or "feeling," it's a systematic, data-based process.

The matrix works like this:

You identify your critical requirements
The matrix suggests 2-3 options that fit
You evaluate specific trade-offs
You make an informed decision

By the end of this capsule, you'll be able to choose a provider for any project in <30 minutes by following the framework.

🎯 Capsule goal

Goal: Apply a structured decision matrix to choose an LLM provider based on prioritized requirements.

By the end of this capsule:

✅ You'll know how to use the decision matrix
✅ You'll be able to map requirements → recommended provider
✅ You'll understand when there are multiple valid options
✅ You'll have clear tie-breaker criteria

📊 The Decision Matrix

Matrix format:

IF your requirements are:
  - Critical Dimension 1: [Value]
  - Critical Dimension 2: [Value]

THEN:
  - Recommended option: [Provider]
  - Why: [Justification]
  - Trade-offs: [What you accept]
  - Alternative: [If option 1 doesn't work]

🎯 Matrix by Requirement Combination

Case 1: Privacy + Cost

Requirements:

Privacy: CRITICAL (sensitive data, on-premise mandatory)
Cost: CRITICAL ($0-500/month)

Recommended provider: Local Ollama

Why:

✅ Privacy: Data NEVER leaves your machine
✅ Cost: $0 operational (only upfront hardware $500-2000)
✅ Compliance: You control 100% (GDPR, HIPAA, any regulation)

Accepted trade-offs:

⚠️ Quality: Mistral 7B/Llama 2 13B are ~80-85% of GPT-3.5
⚠️ Complexity: Initial setup 2-4 hours, you need Linux/CLI skills
⚠️ Speed: 5-10s latency (vs 1-2s OpenAI) if hardware is modest

Alternative: LM Studio (if you prefer a GUI vs CLI)

Real example:

Law firm processing confidential contracts
Hospital analyzing medical records
Bank with PII (personally identifiable information)

Case 2: Quality + Speed

Requirements:

Quality: CRITICAL (accuracy 95%+, complex reasoning)
Speed: CRITICAL (latency <2s)

Recommended provider: OpenAI API (GPT-4 or GPT-3.5-turbo)

Why:

✅ Quality: GPT-4 is state-of-the-art (99%+ accuracy on complex tasks)
✅ Speed: GPT-3.5-turbo ~1.5s, GPT-4 ~3s
✅ Infrastructure: OpenAI has a global CDN, low latency worldwide
✅ Maintenance: Zero (managed service)

Accepted trade-offs:

⚠️ Cost: $0.03/1k tokens (GPT-4) can be $500-2000/month at scale
⚠️ Privacy: Data goes to US cloud (30-day retention)
⚠️ Vendor lock-in: You depend on OpenAI (mitigate with an OpenRouter fallback)

Alternative: OpenRouter (same access to GPT-4, more flexibility)

Real example:

Enterprise customer support (critical accuracy, high volume)
Automated financial analysis
Legal document review (tier 1 quality needed)

Case 3: Simplicity + Cost

Requirements:

Simplicity: CRITICAL (junior team, tight deadline)
Cost: IMPORTANT ($0-200/month)

Recommended provider: OpenRouter (economical models)

Why:

✅ Simplicity: OpenAI-compatible API (drop-in replacement)
✅ Cost: Models from $0.001/1k tokens (20x cheaper than GPT-4)
✅ Flexibility: Change models without refactoring
✅ Setup: <1 hour (pip install, API key, done)

Accepted trade-offs:

⚠️ Quality: Cheap models are 70-80% of GPT-4
⚠️ Speed: Variable by model (2-5s latency)
⚠️ Privacy: Depends on the model (some pass through OpenRouter servers)

Alternative: OpenAI API with GPT-3.5-turbo (if $200/month is enough)

Real example:

Startup in the prototype phase (quick MVP)
Personal side project (no budget)
Agency querying multiple models (A/B testing)

Case 4: Scalability + Speed

Requirements:

Speed: CRITICAL (latency <1s, high throughput)
Scalability: CRITICAL (spikes 10-100x normal traffic)

Recommended provider: Modal (serverless)

Why:

✅ Scalability: Autoscaling 0→1000 instances in seconds
✅ Speed: Warm instances ~1s, cold start ~3s (optimizable)
✅ Cost: Pay-per-use (you don't pay for idle time)
✅ Infrastructure: Modal manages GPUs, containers, networking

Accepted trade-offs:

⚠️ Complexity: You need to learn Modal decorators (1-2 day curve)
⚠️ Cold starts: First requests ~3s (mitigate with warm pools)
⚠️ Variable cost: Can be unpredictable with sudden spikes

Alternative: OpenAI API (also autoscales, but vendor lock-in)

Real example:

A viral app that can have 100x spikes (Product Hunt launch, TikTok)
E-commerce with Black Friday (predictable but massive traffic)
Public API with a strict SLA (<1s p95 latency)

Case 5: Privacy + Quality

Requirements:

Privacy: CRITICAL (on-premise mandatory)
Quality: CRITICAL (accuracy 90%+)

Recommended provider: Ollama with Llama 2 70B (or Mixtral 8x7B)

Why:

✅ Privacy: 100% local, data never leaves
✅ Quality: Llama 2 70B is ~90-92% of GPT-3.5 (acceptable for most tasks)
✅ Operational cost: $0 (only upfront hardware ~$3000-5000 for a GPU)
✅ Control: You manage updates, the model, security

Accepted trade-offs:

⚠️ Hardware: You need a powerful GPU (A100, 4090 Ti, or M2 Ultra Mac)
⚠️ Complexity: Advanced setup (Docker, GPU drivers, networking)
⚠️ Maintenance: Manual updates, 24/7 monitoring if in production

Alternative: LM Studio (if you prefer a GUI and your hardware is enough)

Real example:

Government processing classified documents
Healthcare with strict HIPAA data
Enterprise with extreme compliance (banking, defense)

Case 6: Quick Prototype (MVP)

Requirements:

Simplicity: CRITICAL (launch in 1-3 days)
All others: SECONDARY (you'll optimize later)

Recommended provider: OpenAI API (GPT-3.5-turbo)

Why:

✅ Simplicity: Official SDK, excellent docs, examples everywhere
✅ Speed: ~1.5s latency (good for a demo)
✅ Quality: 85-90% (enough to validate an idea)
✅ Cost: ~$10-50 for an MVP (low volume)

Accepted trade-offs:

⚠️ Vendor lock-in: Doesn't matter in the MVP phase (refactor later if needed)
⚠️ Cost at scale: If the MVP works, you'd migrate after validation
⚠️ Privacy: OK for an MVP (there are no real users yet)

Alternative: LM Studio (if you have a decent laptop and want it free)

Real example:

Hackathon (48 hours for a functional demo)
Pitch to investors (you need a working prototype, not production)
Validate an idea before investing in infra

Case 7: Multi-Provider Flexibility

Requirements:

Flexibility: CRITICAL (being able to change providers without refactoring)
Cost: IMPORTANT (optimize by choosing the cheapest model)

Recommended provider: OpenRouter

Why:

✅ Flexibility: 100+ models (GPT-4, Claude, Gemini, Llama, Mistral)
✅ Cost: Compares prices automatically, picks the cheapest
✅ Fallback: If a model fails, switch to another without code
✅ Compatible API: Drop-in replacement for the OpenAI SDK

Accepted trade-offs:

⚠️ Middleman: OpenRouter between you and the model (adds latency ~100-200ms)
⚠️ Privacy: Some models pass through OpenRouter (read the policies)
⚠️ Rate limits: Different per model

Alternative: Your own unified client (Module 8 of this guide)

Real example:

Agency offering chatbots to clients (each client picks the model)
Startup optimizing costs (starts with GPT-4, then migrates to Mistral)
Developer comparing models (A/B testing)

🔄 The Decision Process (4 Steps)

Step 1: Identify CRITICAL dimensions (2-3 max)

Use the scorecard from capsule 02:

Of the 5 dimensions, which are CRITICAL (non-negotiable)?

[ ] Cost
[ ] Quality
[ ] Privacy
[ ] Speed
[ ] Simplicity

Mark 2-3 max. If you mark 5, none are really critical.

Step 2: Look it up in the matrix

Find the combination that matches your critical dimensions:

My CRITICAL dimensions: Privacy + Cost

I look in the matrix...
→ Case 1: Privacy + Cost → Local Ollama

My CRITICAL dimensions: Quality + Speed

I look in the matrix...
→ Case 2: Quality + Speed → OpenAI API

Step 3: Evaluate trade-offs

The matrix tells you what you accept in exchange:

Recommended option: Local Ollama

Trade-offs:
- ⚠️ Quality: 80-85% vs 95% GPT-4
- ⚠️ Complexity: Setup 2-4 hrs vs <1hr OpenAI
- ⚠️ Speed: 5-10s vs 1-2s OpenAI

Are they acceptable?
- If privacy is CRITICAL (medical data) → YES, trade-offs OK
- If privacy is only "nice-to-have" → NO, use OpenAI

Step 4: Define a fallback plan

Always have a plan B:

Primary option: OpenAI API (GPT-3.5)

Fallback if:
- OpenAI goes down (outage) → OpenRouter with gpt-3.5-turbo
- OpenAI too expensive at scale → Migrate to Ollama Mistral 7B
- Privacy requirements change → Local Ollama from day 1

Write the plan BEFORE implementing.

🎯 Tie-Breaker Criteria

What happens if 2 options are equally good?

Criterion 1: Time-to-market

If tied, choose the simplest:

OpenAI API: <1 day
OpenRouter: <1 day
Modal: 1-2 days
Local Ollama: 2-4 days

Winner: OpenAI API or OpenRouter

Criterion 2: Cost over 12 months

Calculate TCO (Total Cost of Ownership):

OpenAI API:
- Operational cost: $500/month × 12 = $6000
- Setup cost: $0
- Maintenance cost: $0 (managed)
- TOTAL: $6000/year

Local Ollama:
- Operational cost: $0
- Setup cost: $2000 (GPU)
- Maintenance cost: $100/month × 12 = $1200 (DevOps time)
- TOTAL: $3200/year

If volume is constant → Ollama cheaper over 12 months
If volume is low (first months) → OpenAI cheaper in the short term

Winner: Depends on volume and timeframe

Criterion 3: Future flexibility

Which option gives you more options later?

OpenAI API:
- Vendor lock-in: High
- Migration to another provider: Refactoring needed
- Flexibility: Low

OpenRouter:
- Vendor lock-in: Low (change models without code)
- Migration: Easy (compatible API)
- Flexibility: High

Local Ollama:
- Vendor lock-in: Zero (you have the model)
- Migration: N/A (you're already independent)
- Flexibility: Maximum

Winner: Local Ollama (more independence) or OpenRouter (more cloud options)

Criterion 4: Team skills

Can your team maintain it?

Junior Python team:
→ OpenAI API (managed, simple SDK)

Mid team with some DevOps:
→ OpenRouter or Modal (intermediate complexity)

Senior DevOps + ML team:
→ Local Ollama (full control, but requires skills)

Winner: Match your team's skills

💡 Real Use Cases (Applying the Matrix)

Real Case 1: SaaS Startup (Support chatbot)

Requirements:

1000 users/month (growing)
Needs <2s latency
Budget: $200/month initially
Team: 2 junior devs
Data: Non-sensitive (public support)

Applying the matrix:

CRITICAL dimensions: Simplicity + Speed

I look in the matrix... There's no exact match for "Simplicity + Speed"

Candidate options:

Case 2 (Quality + Speed) → OpenAI API
Case 3 (Simplicity + Cost) → OpenRouter

Evaluation:

OpenAI API (GPT-3.5):

✅ Simplicity: Setup <1 day
✅ Speed: ~1.5s (meets <2s)
⚠️ Cost: 1000 users × 10 queries × 500 tokens = 5M tokens/month = $7.50/month
COST OK (within $200/month)

OpenRouter (Mixtral):

✅ Simplicity: Setup ~1 day (similar to OpenAI)
✅ Speed: ~2-3s (borderline <2s)
✅ Cost: ~$3/month (cheaper)

Decision: OpenAI API

Meets the speed target more comfortably
$7.50/month is negligible
Better docs for a junior team

Fallback: OpenRouter if OpenAI raises prices or latency degrades

Real Case 2: Healthcare App (Symptom analysis)

Requirements:

500 patients/day
Medical data (HIPAA compliance)
Accuracy 95%+ (lives at stake)
Budget: No limit (large hospital)
Team: In-house senior DevOps

Applying the matrix:

CRITICAL dimensions: Privacy + Quality

I look in the matrix... → Case 5: Privacy + Quality → Ollama with Llama 2 70B

Evaluation:

Local Ollama Llama 2 70B:

✅ Privacy: On-premise, HIPAA compliant
✅ Quality: 90-92% (acceptable with human oversight)
✅ Cost: $5k upfront (hardware), then $0
✅ Skills: A senior DevOps team can maintain it

Trade-offs:

⚠️ Quality: 90% vs 99% GPT-4
- Mitigation: Human-in-the-loop (a doctor reviews suggestions)
⚠️ Complexity: Setup 1 week
- Acceptable: No tight deadline, security > speed

Decision: Local Ollama

Privacy is NON-negotiable (HIPAA)
Quality with human oversight is enough
Cost is not a problem

Fallback: NONE (privacy is a hard requirement, it can't go to the cloud)

Future alternative: If the FDA approves GPT-4 with guaranteed data residency

Real Case 3: E-commerce (Personalized recommendations)

Requirements:

50k users/day
Black Friday spikes (10x traffic)
Latency <1s (critical UX)
Budget: $2000/month
Team: Mid-level DevOps

Applying the matrix:

CRITICAL dimensions: Scalability + Speed

I look in the matrix... → Case 4: Scalability + Speed → Modal serverless

Evaluation:

Modal serverless (Mistral 7B):

✅ Scalability: 0→1000 instances automatic
✅ Speed: ~1-2s (warm), ~3s (cold start)
✅ Cost: Pay-per-use, ~$1500/month normal, $15k Black Friday (1-day spike)
- Annualized cost: ($1500 × 11) + $15k = $31.5k/year = $2625/month on average
- ⚠️ EXCEEDS BUDGET ($2000/month)

OpenAI API (GPT-3.5):

✅ Scalability: Automatic
✅ Speed: ~1.5s
✅ Cost: 50k × 5 queries × 200 tokens × 30 = 1.5B tokens/month = $3k normal
- Black Friday (10x): $30k that month
- Annualized: ($3k × 11) + $30k = $63k/year = $5,250/month on average
- ⚠️ ALSO EXCEEDS BUDGET

Re-evaluation:

A $2000/month budget is NOT realistic for 50k users/day with spikes.

Options:

Increase the budget to $3-4k/month (justifiable with revenue)
Local Ollama with a GPU cluster ($10k upfront, then free)
- Meets the cost target in the long run
- But complexity is high for a mid-level team
Hybrid: OpenAI normally, Ollama for Black Friday (pre-cache)

Decision: Increase the budget + OpenAI API

$3k/month is 0.3% of typical e-commerce revenue
Operational simplicity > marginal savings
The Black Friday peak is 1 day (a cost spike is acceptable)

Fallback: If the budget is NOT approved → Local Ollama (requires hiring a senior DevOps)

✅ Decision Checklist

Use this checklist BEFORE implementing:

## Final Decision Checklist

### 1. Clear requirements
- [ ] I identified 2-3 CRITICAL dimensions
- [ ] I quantified the values ($/month, latency ms, accuracy %)
- [ ] I validated with stakeholders (product, legal, finance)

### 2. Chosen option
- [ ] I applied the decision matrix
- [ ] I'll evaluate the specific trade-offs
- [ ] The trade-offs are acceptable given the context

### 3. Fallback plan
- [ ] I have an option B if option A fails
- [ ] The fallback is viable (not "we hope it works")
- [ ] Clear trigger criteria ("if latency > 5s for 10 min, switch")

### 4. Technical validation
- [ ] My team has the necessary skills
- [ ] Time-to-market is acceptable
- [ ] The 12-month cost is sustainable

### 5. Compliance and legal
- [ ] Privacy: Reviewed by legal (if sensitive data)
- [ ] Compliance: GDPR/HIPAA/SOC2 verified
- [ ] Contracts: Terms of service acceptable

IF all checks are ✅ → PROCEED TO IMPLEMENT
IF any check is ❌ → RE-EVALUATE the option

📝 Exercise: Apply the Matrix

Exercise 1: Fintech Startup

Context:

Personal financial advisory app
5000 registered users
Needs to analyze transactions (sensitive data)
Budget: $1000/month
Team: 3 mid-level devs
Needs to launch in 2 weeks (investor demo)

Your task:

Identify the CRITICAL dimensions (2-3)
Look up the combination in the matrix
Evaluate the trade-offs
Define a fallback plan

See the recommended answer

Analysis:

CRITICAL dimensions:

Privacy (sensitive financial data)
Simplicity (2-week deadline, mid-level team)

IMPORTANT dimensions: 3. Cost ($1000/month is the limit)

SECONDARY dimensions: 4. Quality (80% OK with human review) 5. Speed (advisory is not real-time, <5s OK)

I look in the matrix:

There's no exact match for "Privacy + Simplicity"

Candidates:

Case 1 (Privacy + Cost) → Local Ollama
Case 5 (Privacy + Quality) → Local Ollama

Problem: Local Ollama has LOW simplicity (setup 2-4 days, skills needed)

Conflict: Privacy CRITICAL but Simplicity also CRITICAL

Options:

Option A: Local Ollama

✅ Maximum privacy
✅ $0 operational cost
❌ Simplicity: Setup 2-4 days (eats into the deadline)
❌ Requires DevOps skills the mid team doesn't quite have

Option B: OpenAI API

❌ Privacy: Financial data to the cloud (legal/compliance risk)
✅ Simplicity: Setup 1 day
✅ Meets the 2-week deadline
⚠️ Cost: ~$500/month (within budget)

Option C: Hybrid (for the demo)

OpenAI API for the investor demo (2 weeks)
Migrate to Ollama after the demo (4-6 weeks)
✅ Meets the deadline
✅ Privacy in the medium term
⚠️ Refactoring needed

Decision: Option C (Hybrid)

Justification:

Short-term (demo): OpenAI API
- The demo data is mock (not real)
- Setup 1 day, works for the pitch
- You mention in the pitch "we'll migrate to on-premise post-funding"
Long-term (post-funding): Local Ollama
- You hire a senior DevOps
- Setup in 4-6 weeks
- Compliance BEFORE real users

Trade-offs:

⚠️ Refactoring cost (~1 week of dev time)
✅ Acceptable because the demo does NOT use real data

Fallback plan:

If investors ask about privacy → You have a clear plan (Ollama post-funding)
If funding isn't secured → Ollama anyway (with time, no deadline)

📊 Summary

Key concepts:

The decision matrix structures the choice:
- It's not ad-hoc or "feeling"
- It maps requirements → provider systematically
- 7 common cases covered
A 4-step process:
- Step 1: Identify 2-3 CRITICAL dimensions
- Step 2: Look up the combination in the matrix
- Step 3: Evaluate the trade-offs
- Step 4: Define a fallback plan
Tie-breaker criteria:
- Time-to-market (the simplest wins)
- 12-month TCO (cheaper in the long run)
- Future flexibility (less vendor lock-in)
- Team skills (match capabilities)
Always have a plan B:
- OpenAI API → Fallback: OpenRouter
- Local Ollama → Fallback: NONE (if privacy is critical)
- Modal → Fallback: OpenAI API

🔗 Additional resources

Decision Framework Template (Google Sheet) - Reusable template
LLM Provider Comparison Tool - Compare specs
TCO Calculator for LLMs - Total cost of ownership
GDPR Compliance Checker - Privacy validation

➡️ Next step

Next capsule: 05-quantitative-trade-offs.md

Now that you have the decision framework, you'll learn to compare options with real quantitative data:

Exact cost ($/1M tokens)
Average latency (benchmarks)
Quality (accuracy scores)
Privacy (compared policies)

The numbers will help you validate your matrix decision.

Reading time: 12-15 minutes
Next: 05-quantitative-trade-offs.md

← PreviousThe Options Landscape 2024-2026: LLM Access Providers Next →Quantitative Trade-offs: A Comparison with Real Data