Module 1: Decision Framework for LLM Access

The 5 Evaluation Dimensions for Choosing an LLM Provider

Next →

Capsule overview

To choose the optimal LLM provider, you need to evaluate your project against 5 critical dimensions: Cost, Quality, Privacy, Speed, and Simplicity.

Each dimension has trade-offs. There is no universal "best option." The OpenAI API can be perfect for one project (maximum quality, simplicity) but terrible for another (prohibitive cost, unacceptable privacy). Local Ollama can be ideal for one (free, private) but inadequate for another (complex setup, expensive hardware).

This capsule teaches you to systematically evaluate these 5 dimensions for any project. By the end, you'll have a scorecard that tells you which provider best fits your specific requirements.

🎯 Capsule goal

Goal: Be able to evaluate any project against the 5 dimensions and prioritize which are critical vs nice-to-have.

By the end of this capsule:

  • ✅ You'll understand what to evaluate in each dimension
  • ✅ You'll know how to ask the key questions to extract requirements
  • ✅ You'll be able to prioritize dimensions by context
  • ✅ You'll have a basic scorecard to use in capsule 08

📊 Dimension 1: Cost

What to evaluate?

Cost is NOT just "$/month". It includes:

  1. Cost per use (pay-per-use)

    • $/1M tokens (input + output)
    • $/request
    • $/compute hour

  2. Upfront costs (one-time)

    • Hardware (GPU, RAM for local)
    • Licenses (if applicable)
    • Initial setup (team time)

  3. Hidden costs

    • Maintenance (updates, monitoring)
    • Scaling (what happens with 10x users)
    • Downtime (if the provider goes down, do you lose sales?)

Key questions:

1. What is my monthly budget? ($0, $50, $500, $5000, unlimited)
2. How many users/requests do I expect? (100/day, 10k/day, 1M/day)
3. Do I have budget for initial hardware? (Can I buy a $2000 GPU?)
4. Is the cost predictable or variable? (Do I need to know how much I'll spend each month?)

Examples per provider:

ProviderCost per useUpfrontScalability
OpenAI API$0.0015/1k tokens (GPT-3.5)
$0.03/1k tokens (GPT-4)		$0 (free to start)Linear (2x users = 2x cost)
Local Ollama$0 (always free)$500-2000 (hardware)Fixed (same cost for 100 or 10k users)
Modal serverless$0.000185/s of GPU$0 (pay-per-use)Automatic (scales without intervention)
OpenRouterVariable ($0.001-0.03 depending on model)$0 (free to start)Linear

How to prioritize:

Cost is CRITICAL if:

  • ✅ Startup with no funding
  • ✅ Personal project/side project
  • ✅ High volume (millions of requests/month)
  • ✅ Tight profit margin

Cost is SECONDARY if:

  • ⚠️ Company with a budget
  • ⚠️ Prototype (premature optimization)
  • ⚠️ B2B with large contracts (cost is passed on to the client)

Evaluation tip:

Estimate your monthly cost:

Monthly cost = (requests/day) × (tokens/request) × ($/token) × 30 days

Example:
- 10,000 requests/day
- 500 tokens/request on average
- $0.002/1k tokens (GPT-3.5)
- 30 days

Cost = 10k × 500 × ($0.002/1000) × 30 = $300/month

If your calculation gives >$500/month, consider local options or serverless with cost optimization.

🏆 Dimension 2: Quality

What to evaluate?

Quality is NOT just "how good the model is". It includes:

  1. Model capabilities

    • Reasoning (logic, math)
    • Knowledge (what it knows)
    • Supported languages
    • Context window (how much text it can process)

  2. Accuracy

    • How correct is it? (90%, 95%, 99%?)
    • Does it hallucinate? (makes up information)
    • Does it follow instructions? (if you say "only answer YES/NO", does it?)

  3. Consistency

    • Does it give the same answer to the same question?
    • Does its "personality" change?

Key questions:

1. How critical is accuracy? (Technical support: 95%+. Casual chatbot: 80% OK.)
2. Do I need complex reasoning? (Legal analysis: YES. Simple translation: NO.)
3. What context window do I need? (Long documents: 32k+. Short chat: 4k OK.)
4. Can I tolerate hallucinations? (Medical data: NO. Creative ideas: Yes.)

Examples per provider/model:

ModelReasoningContextHallucinationsBest for
GPT-4 (OpenAI)Excellent128k tokensLowComplex tasks, analysis
GPT-3.5 (OpenAI)Good16k tokensMediumChatbots, support
Claude 3 (OpenRouter)Excellent200k tokensVery lowLong documents
Mistral 7B (local Ollama)Moderate8k tokensMedium-HighPrototypes, dev
Llama 2 70B (Local)Good4k tokensMediumLocal/quality balance

How to prioritize:

Quality is CRITICAL if:

  • ✅ Business decisions (finance, legal, medical)
  • ✅ Customer-facing (brand reputation)
  • ✅ Compliance/regulatory (you can't give incorrect answers)

Quality is SECONDARY if:

  • ⚠️ Internal prototype (just to test an idea)
  • ⚠️ Simple tasks (translation, basic summary)
  • ⚠️ You can validate manually afterward (human-in-the-loop)

Evaluation tip:

Quick quality test:

  1. Write 5 questions representative of your use
  2. Test with GPT-4 (high baseline)
  3. Test with a cheaper model (GPT-3.5, Mistral 7B)
  4. Does the quality difference justify the cost difference?

If GPT-3.5 is 95% as good as GPT-4 but 20x cheaper → use GPT-3.5

🔒 Dimension 3: Privacy

What to evaluate?

Privacy is NOT just "is my data stored?". It includes:

  1. Data retention

    • How long do they keep your requests?
    • Do they use them to train models?
    • Can humans read them?

  2. Compliance (regulatory compliance)

    • GDPR (Europe)
    • HIPAA (US healthcare)
    • CCPA (California)
    • SOC 2, ISO 27001

  3. Data location

    • Where are the servers?
    • Do they cross borders?
    • On-premise vs cloud?

Key questions:

1. Is my data sensitive? (Medical, financial, legal, personal)
2. Do I have legal constraints? (GDPR, HIPAA, client contracts)
3. Can I send data to the US? (If you're in the EU, this can be a legal issue)
4. Do I need an audit trail? (Must I be able to prove data never left?)

Examples per provider:

ProviderData retentionTrainingOn-premiseCompliance
OpenAI API30 days (can change)NO (opt-out by default since 2023)NO (US cloud)SOC 2
Local OllamaNEVER (all local)NEVERYES (your hardware)You control it
ModalDuring executionNONO (cloud)SOC 2
OpenRouterDepends on the modelDepends on the modelNOVariable

How to prioritize:

Privacy is CRITICAL if:

  • ✅ Medical data (HIPAA)
  • ✅ Financial data (PCI-DSS)
  • ✅ Legal data (attorney-client privilege)
  • ✅ Personal data (strict GDPR)
  • ✅ Contracts with enterprise clients (confidentiality clauses)

Privacy is SECONDARY if:

  • ⚠️ Public data (FAQ, public documentation)
  • ⚠️ Anonymized data
  • ⚠️ Prototype without real data

Evaluation tip:

Privacy test:

Ask yourself: What happens if this data appears in the New York Times tomorrow?

A. "Nothing, it's public" → Privacy secondary
B. "We lose clients" → Privacy important
C. "We go to jail" → Privacy CRITICAL (use local)

If the answer is C, ONLY on-premise options (local Ollama, LM Studio).

⚡ Dimension 4: Speed

What to evaluate?

Speed is NOT just "how fast it responds". It includes:

  1. Latency (response time)

    • Time to first token (TTFT): How long it takes to start responding
    • Tokens/second: Generation speed
    • End-to-end: Total request time

  2. Throughput (volume)

    • Simultaneous requests
    • Total tokens/minute of the system

  3. Consistency

    • Is latency predictable? (always 2s vs 1s-10s)
    • Does it degrade under load?

Key questions:

1. What is an acceptable latency? (Chatbot: <2s. Batch analysis: <1min OK.)
2. How many simultaneous requests? (1 user: 1. Web app: 100+.)
3. Do I need streaming? (Watching the response generate word by word)
4. Do I have traffic spikes? (Black Friday, viral moments, events)

Examples per provider:

ProviderTypical latencyThroughputStreamingScalability
OpenAI API1-3s (GPT-3.5)
3-5s (GPT-4)		High (rate limits: 3500 req/min tier 1)✅ YesAutomatic
Local Ollama3-10s (depends on hardware)Low (1 request at a time)✅ YesManual (more hardware)
Modal serverless2-4s (cold start)
1-2s (warm)		High (autoscaling)✅ YesAutomatic
LM Studio5-15s (CPU)
2-5s (GPU)		Low (1-2 parallel)✅ YesNo (desktop tool)

How to prioritize:

Speed is CRITICAL if:

  • ✅ Real-time chat (users waiting)
  • ✅ Customer support (every second counts)
  • ✅ Interactive apps (IDE assistants, live demos)

Speed is SECONDARY if:

  • ⚠️ Batch processing (nightly log analysis)
  • ⚠️ Background tasks (generating reports)
  • ⚠️ Internal admin tools (not customer-facing)

Evaluation tip:

Basic benchmark:

Test with a typical prompt (200 tokens output):

OpenAI GPT-3.5: ~2s
Ollama Mistral 7B (local M1): ~6s
Modal (cold start): ~4s
Modal (warm): ~2s

Is a 4s difference acceptable?
- Chatbot: NO
- Daily report: YES

If you need <2s consistently, the OpenAI API or Modal warm are better options.

🛠️ Dimension 5: Simplicity

What to evaluate?

Simplicity is NOT just "easy to use". It includes:

  1. Setup time

    • How long until it's up and running?
    • Do I need special infrastructure?

  2. Technical complexity

    • Do I need specific skills? (DevOps, GPU tuning)
    • Can I do it with my current team?

  3. Maintenance

    • Do I need to update/patch?
    • Do I need to monitor 24/7?
    • What happens if something breaks?

Key questions:

1. How much time do I have to implement? (1 day, 1 week, 1 month)
2. What skills does my team have? (Only Python, advanced DevOps, ML engineers)
3. Who maintains this afterward? (Do I have time/team for operations?)
4. Do I need documentation in Spanish? (OpenAI docs: excellent. Ollama: fewer.)

Examples per provider:

ProviderSetup timeSkills neededMaintenanceDocs
OpenAI API<1 hourBasic PythonZero (managed)Excellent
Local Ollama2-4 hoursLinux, CLI, networkingMedium (updates, monitoring)Good
Modal serverless1-2 hoursPython + decoratorsLow (almost managed)Good
LM Studio<30 minClick & playLow (auto-update)Intuitive GUI
OpenRouter<1 hourBasic PythonZero (managed)Good

How to prioritize:

Simplicity is CRITICAL if:

  • ✅ Junior team (no DevOps)
  • ✅ Tight deadline (MVP in days)
  • ✅ No time for operations (small startup)
  • ✅ You want to focus on the product (not infrastructure)

Simplicity is SECONDARY if:

  • ⚠️ Senior team (in-house DevOps)
  • ⚠️ Time available (you can invest weeks)
  • ⚠️ Requirements that force complexity (on-premise privacy)

Evaluation tip:

Simplicity test:

Estimated time from "zero to hello world":

OpenAI API:
- Create account: 5 min
- pip install openai: 1 min
- Hello world code: 5 min
- Total: ~15 min ✅

Local Ollama:
- Install Ollama: 10 min
- Download model (7GB): 20 min
- Networking setup: 15 min
- Hello world code: 10 min
- Total: ~55 min ⚠️

Is 40 extra minutes worth it to have it local?
- If privacy is critical: YES
- If you just want to prototype: NO

🎯 Applying the 5 Dimensions: Scorecard

Evaluation template:

Use this scorecard for any project:

## Project: [Name]

### 1. Cost
- Monthly budget: $____
- Expected volume: ____ requests/day
- Do I have hardware?: YES / NO
- Priority: CRITICAL / IMPORTANT / SECONDARY

### 2. Quality
- Accuracy needed: ____%
- Do I need complex reasoning?: YES / NO
- Context window: ____ tokens
- Priority: CRITICAL / IMPORTANT / SECONDARY

### 3. Privacy
- Data type: SENSITIVE / NORMAL / PUBLIC
- Compliance: GDPR / HIPAA / NONE
- Can it go to the cloud?: YES / NO
- Priority: CRITICAL / IMPORTANT / SECONDARY

### 4. Speed
- Acceptable latency: ____ seconds
- Simultaneous requests: ____
- Do I need streaming?: YES / NO
- Priority: CRITICAL / IMPORTANT / SECONDARY

### 5. Simplicity
- Time available: ____ days
- Team skills: JUNIOR / MID / SENIOR
- Who maintains it?: NOBODY / SMALL TEAM / DEDICATED TEAM
- Priority: CRITICAL / IMPORTANT / SECONDARY

Worked example:

## Project: E-commerce Chatbot

### 1. Cost
- Monthly budget: $500
- Expected volume: 10,000 requests/day
- Do I have hardware?: NO
- Priority: IMPORTANT (not critical, but $500 is the limit)

### 2. Quality
- Accuracy needed: 90%
- Do I need complex reasoning?: NO (basic support)
- Context window: 4k tokens (short conversations)
- Priority: IMPORTANT (90% is enough, I don't need GPT-4)

### 3. Privacy
- Data type: NORMAL (public support, no cards)
- Compliance: NONE
- Can it go to the cloud?: YES
- Priority: SECONDARY

### 4. Speed
- Acceptable latency: 2 seconds
- Simultaneous requests: 50
- Do I need streaming?: YES (better UX)
- Priority: CRITICAL (users won't wait >2s)

### 5. Simplicity
- Time available: 3 days (quick MVP)
- Team skills: JUNIOR (2 basic-Python devs)
- Who maintains it?: NOBODY (no DevOps)
- Priority: CRITICAL (junior team, tight deadline)

---

SUMMARY:
CRITICAL dimensions: Speed, Simplicity
IMPORTANT dimensions: Cost, Quality
SECONDARY dimensions: Privacy

PRELIMINARY RECOMMENDATION: OpenAI API (GPT-3.5)
- ✅ Meets speed (<2s)
- ✅ Meets simplicity (setup in 1 day)
- ✅ Meets cost ($300/month < $500 limit)
- ✅ Meets quality (90%+ with GPT-3.5)
- ✅ Privacy OK (public data)

🔄 Prioritizing Trade-offs

Golden rule:

Optimize for the 2-3 CRITICAL dimensions. Accept trade-offs on the rest.

You can't max out all 5 dimensions:

  • OpenAI API: High quality, high simplicity → Medium cost, medium privacy
  • Local Ollama: Zero cost, maximum privacy → Medium quality, high complexity
  • Modal serverless: High scalability, medium simplicity → Variable cost

Quick decision matrix:

If your 2 CRITICAL dimensions are:

1. COST + PRIVACY → Local Ollama
2. QUALITY + SPEED → OpenAI API
3. SIMPLICITY + SPEED → OpenAI API or OpenRouter
4. COST + SIMPLICITY → OpenRouter (cheap models)
5. PRIVACY + QUALITY → Local with Llama 2 70B (expensive hardware)

In the next capsule (03) you'll see the complete landscape of options with specific comparisons.

📝 Application exercise

Exercise: Evaluate these 3 projects

For each project, identify:

  1. Which 2-3 dimensions are CRITICAL
  2. Which provider you would preliminarily recommend

Project A: Legal contract analysis

  • 1000 contracts/month
  • Each contract: 10k tokens on average
  • Law firm, confidential data
  • Needs 99%+ accuracy
  • A team of 1 senior dev

Project B: Creative ideas chatbot

  • Personal side project
  • 100 users/month
  • No budget ($0)
  • 70% accuracy OK (just ideas, not critical)
  • Just you (junior Python dev)

Project C: Enterprise technical support

  • 100k requests/day
  • Needs <1s latency
  • Non-sensitive data
  • Budget: $5000/month
  • DevOps team available
See the recommended answers

Project A: Legal analysis

CRITICAL dimensions:

  1. PRIVACY (confidential data, attorney-client privilege)
  2. QUALITY (99%+ accuracy, legal decisions)

IMPORTANT dimensions: 3. Cost (1000 × 10k tokens = 10M tokens/month = $200-600 depending on the model)

Recommendation: Local Ollama with Llama 2 70B or LM Studio

  • ✅ Maximum privacy (on-premise, data never leaves)
  • ✅ High quality (Llama 2 70B is almost GPT-3.5)
  • ✅ Cost: Only hardware (~$3k upfront, then free)
  • ⚠️ Trade-off: Complexity (a senior dev can handle it)

Project B: Creative chatbot

CRITICAL dimensions:

  1. COST (budget $0)
  2. SIMPLICITY (junior dev alone)

SECONDARY dimensions:

  • Quality (70% OK)
  • Speed (side project, not critical)
  • Privacy (ideas not sensitive)

Recommendation: Local Ollama with Mistral 7B

  • ✅ Cost: $0 (always free)
  • ✅ Simplicity: Medium (but acceptable for learning)
  • ✅ Quality: 70-80% (enough for ideas)
  • ⚠️ Trade-off: Initial setup (2-4 hours the first time)

Alternative if you have a weak laptop: OpenRouter with free models (some available)

Project C: Enterprise support

CRITICAL dimensions:

  1. SPEED (<1s latency, 100k req/day)
  2. SCALABILITY (traffic spikes)

IMPORTANT dimensions: 3. Cost ($5k/month available)

Recommendation: OpenAI API (GPT-3.5-turbo) with an OpenRouter fallback

  • ✅ Speed: ~1-2s (meets <1s at the 90th percentile)
  • ✅ Scalability: Automatic (OpenAI handles the traffic)
  • ✅ Cost: 100k × 500 tokens × $0.002/1k × 30 = $3000/month (within budget)
  • ✅ Fallback: OpenRouter if OpenAI goes down
  • ⚠️ Trade-off: Vendor lock-in (mitigated with a fallback)

Alternative if <1s is strict: Modal serverless (warm instances <1s)

📊 Summary

Key concepts:

  1. 5 Evaluation dimensions:

    • Cost ($/month, upfront, hidden)
    • Quality (reasoning, accuracy, context)
    • Privacy (retention, compliance, location)
    • Speed (latency, throughput, consistency)
    • Simplicity (setup, skills, maintenance)

  2. There is no "best" provider:

    • There is only "the best FOR your project"
    • Identify 2-3 CRITICAL dimensions
    • Accept trade-offs on the rest

  3. Use a structured scorecard:

    • Evaluate all dimensions
    • Prioritize (CRITICAL / IMPORTANT / SECONDARY)
    • The preliminary recommendation emerges naturally

  4. Trade-offs are inevitable:

    • OpenAI: Quality + Simplicity → Medium cost
    • Ollama: Zero cost + Privacy → High complexity
    • Modal: Scalability → Variable cost

🔗 Additional resources

  1. OpenAI Pricing Calculator - Estimate costs
  2. LLM Comparison Leaderboard - Compare model quality
  3. GDPR Compliance Guide - Europe's privacy requirements
  4. LLM Latency Benchmarks - Speed comparison
  5. AWS vs Local Cost Analysis - Economic trade-offs

➡️ Next step

Next capsule: 03-the-options-landscape-2024-2026.md

Now that you know HOW to evaluate (5 dimensions), you'll learn WHAT options exist:

  • OpenAI API (standard cloud)
  • LM Studio (local GUI)
  • Ollama (local CLI)
  • OpenRouter (aggregator)
  • Modal (serverless)

You'll see the main features, trade-offs, and when to use each one.

Reading time: 10-12 minutes
Next: 03-the-options-landscape-2024-2026.md

← PreviousModule 1: Decision Framework for LLM AccessNext →The Options Landscape 2024-2026: LLM Access Providers