Module 7: Technical comparison of providers
The full comparison table
You've reached the quick reference capsule. Here we consolidate everything learned in the module (and the whole path so far) into a single view. It's not a capsule to read linearly — it's a capsule to bookmark and return to when you have to decide or explain.
By the end you'll be able to:
- Have on hand a master table of the 5 providers across all dimensions
- Recognize in seconds which provider is best for which case
- Share the table with your team or cofounder as a common reference
- Adapt the table to your product by adding or removing dimensions according to your context
The master table
Data as of early 2026. The numbers change — use this table as a structural template, not as eternal truth. Replace with your real benchmarks for production.
Executive summary
| Provider | Type | Setup | Typical P95 latency | Cost /M tokens | Typical quality | When it wins |
|---|---|---|---|---|---|---|
| OpenAI | Managed cloud | 5 min | 1.5-3s | $0.15-$2.50 input | ⭐⭐⭐⭐⭐ | Safe default, MVP, high quality |
| Anthropic | Managed cloud | 5 min | 1.8-3.5s | $3-$15 input | ⭐⭐⭐⭐⭐ | Complex reasoning, long contexts |
| OpenRouter | Aggregator | 5 min | 2-4s | $0.07-$15 (variable) | ⭐⭐⭐⭐ | Multi-model, fallback, low cost with OSS |
| LM Studio | Local (GUI) | 10 min | 4-8s (local CPU/GPU) | $0 operational | ⭐⭐⭐ | Local dev, privacy, no API key |
| Ollama | Local (CLI) | 15 min | 3-7s (varies by hardware) | $0 operational | ⭐⭐⭐ | On-prem production, Docker integration |
| Modal | Serverless GPU | 20 min | 2-4s (warm) / 15-30s (cold) | GPU-second | ⭐⭐⭐ (OSS model) to ⭐⭐⭐⭐⭐ (if you put a GPT-equivalent) | Variable traffic, OSS models without DevOps |
By technical dimension
| Dimension | OpenAI | Anthropic | OpenRouter | LM Studio | Ollama | Modal |
|---|---|---|---|---|---|---|
| Setup | 5min | 5min | 5min | 10min | 15min | 20min |
| Hosting | US / multi | US | US / EU configurable | Local | Local / VPS | Multi-region |
| Auth | API key | API key | API key | No auth (local) | No auth (local) | Bearer token |
| Available models | ~10 (closed) | ~6 (closed) | 100+ (open + closed) | Open (HF) | Open (curated) | Any (open) |
| Streaming | ✅ SSE | ✅ SSE | ✅ SSE | ✅ SSE | ✅ SSE | ✅ with your impl |
| Function calling | ✅ Mature | ✅ Mature | ⚠️ Depends on the model | ⚠️ Limited | ⚠️ Limited | ✅ with your impl |
| Strict JSON mode | ✅ | ✅ | ⚠️ Depends on the model | ⚠️ Limited | ⚠️ Limited | ✅ with your impl |
| Multimodal (vision) | ✅ gpt-4o | ✅ Claude 3+ | ⚠️ Some models | ⚠️ Some | ⚠️ Some (llava) | ✅ with your impl |
| Vendor lock-in | Medium-high | Medium-high | Low (multi-model) | Low | Low | Low (your code) |
By economic dimension
| Dimension | OpenAI | Anthropic | OpenRouter | LM Studio | Ollama (self-host) | Modal |
|---|---|---|---|---|---|---|
| Cost model | Per token | Per token | Per token | $0 + hardware | $0 + hardware/VM | GPU-second |
| Initial free tier | Limited trial | Credits | Credits | N/A | N/A | $30/month |
| Cost for 10K req/month | ~$5-50 | ~$15-100 | ~$1-30 | $0 | $0 (local) | ~$10-30 |
| Cost for 1M req/month | ~$500-5000 | ~$1500-15000 | ~$100-3000 | $0 (doesn't scale this way) | ~$720 (1 GPU) | ~$300-1500 |
| Warm pool / fixed | N/A | N/A | N/A | N/A | Yes: VM 24/7 | Optional with min_containers |
| Cost predictability | High (per token) | High (per token) | High | N/A | Very high (fixed) | Medium (depends on usage) |
| Cost optimization available | Caching, small models | Caching, prompt batching | Switch to a cheaper model | N/A | Optimize batching | Quantization, GPU sizing |
By operational requirements
| Requirement | OpenAI | Anthropic | OpenRouter | LM Studio | Ollama | Modal |
|---|---|---|---|---|---|---|
| Needs external API key | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ |
| Needs a credit card | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ (after free tier) |
| Works offline | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ |
| DevOps needed | Minimal | Minimal | Minimal | Minimal | Medium-high | Minimal |
| Human maintenance | Near zero | Near zero | Near zero | Low | Medium | Low |
| Autoscaling | ✅ Auto | ✅ Auto | ✅ Auto | ❌ | ❌ (manual) | ✅ Auto |
| Cold starts | No | No | No | No | Only on startup | Yes (mitigable) |
| Enterprise support / SLA | ✅ Plans | ✅ Plans | ⚠️ Limited | ❌ | ❌ | ⚠️ Pro plans |
By compliance / privacy requirements
| Requirement | OpenAI | Anthropic | OpenRouter | LM Studio | Ollama | Modal |
|---|---|---|---|---|---|---|
| SOC2 Type II | ✅ | ✅ | Verify | N/A | N/A | ✅ |
| HIPAA BAA | ✅ (plans) | ✅ (plans) | Verify | ⚠️ You control it | ⚠️ You control it | Verify |
| GDPR | ✅ | ✅ | ✅ | ✅ (local) | ✅ (local) | ✅ |
| EU data residency | ⚠️ Configurable | ⚠️ Configurable | ⚠️ Per provider | ✅ (local) | ✅ (local) | ✅ with EU region |
| No data retention for training | ✅ default | ✅ default | Depends on provider | ✅ | ✅ | ✅ |
| On-premise possible | ❌ | Limited (Bedrock) | ❌ | ✅ | ✅ | ❌ |
The "what I use each one for" table
Quick mapping of use case → primary recommendation + alternatives:
| Use case | Primary recommendation | Alternatives | Why |
|---|---|---|---|
| MVP with a tiny budget ($0) | LM Studio or Ollama local | OpenRouter free tier | No paying and you learn |
| Quick MVP to validate | OpenAI GPT-4o-mini | Anthropic Claude 3.5 Haiku | 5min setup, high quality, low cost at low volume |
| High-volume B2C product | OpenRouter with Mistral/Mixtral | Self-hosted if volume >2M req/month | Low cost, acceptable quality |
| Premium B2B product | OpenAI gpt-4o or Anthropic Claude 3.5 Sonnet | Modal with Mixtral 8x22B | Quality matters more than cost |
| Medical / legal assistant | Anthropic Claude (reasoning) | OpenAI gpt-4o | Precision + reasoning |
| Multi-language chatbot | OpenAI or Anthropic | OpenRouter with a specialized model | Robust multilingual support |
| Nightly batch analysis | OpenRouter with Mistral | Modal with quantization | Low cost, latency doesn't matter |
| Burst traffic (events) | Modal | OpenAI with external autoscaling | Serverless scales automatically |
| Mandatory on-prem privacy | Ollama self-hosted | LM Studio on dedicated machines | Data never leaves |
| Local dev / experimentation | LM Studio (GUI) or Ollama (CLI) | Modal with free tier | No cost, fast iteration |
| Multi-provider fallback | OpenRouter | Your Unified Client (M08) | Automatic failover |
| Edge / IoT with a lightweight LLM | Ollama on device + a small model | LM Studio with phi-3 | Local, no network |
Quick decision diagram
Privacy/on-prem constraint?
├── YES → Ollama or LM Studio (local)
└── NO ↓
Expected monthly volume?
├── < 50K req/month → OpenAI GPT-4o-mini or Anthropic Claude Haiku
├── 50K-500K req/month ↓
│ ├── Quality critical → OpenAI gpt-4o or Anthropic Claude Sonnet
│ ├── Cost critical → OpenRouter Mistral
│ └── Burst traffic → Modal
└── > 500K req/month ↓
├── GPT-4 quality mandatory → OpenAI (more rate limit)
├── OSS acceptable, constant → Self-hosted Ollama
└── OSS acceptable, variable → Modal with autoscaling
How to use this table in a decision conversation
Bad pattern: "Modal is better because it has autoscaling."
Good pattern: "For our traffic of 200K req/month with 3× bursts during business hours, Modal comes out to ~$300/month and OpenAI to ~$200/month. The cost difference is marginal ($100/month); quality decides: with GPT-4o-mini we get 86% on our eval set, with Mistral 7B we get 69%. If you need the 86%, OpenAI; if 69% is enough, Modal gives us model control. I recommend OpenAI given our SLA."
The difference: the second is a technical decision with numbers.
Common traps when using comparison tables
Trap 1 — "I trust a blog's table." Blog tables age. Prices change, models come and go. This table is a template. Replace it with your benchmarks before committing.
Trap 2 — "Little stars are a metric." "⭐⭐⭐⭐ vs ⭐⭐⭐⭐⭐" isn't measurable information. Those little stars in this table are indicative for getting started; your real decision goes with numbers.
Trap 3 — "A single table decides everything." A table is input. The decision includes: team familiarity, enterprise support, legal contracts, the provider's roadmap. The table narrows down, it doesn't decide.
Trap 4 — "I don't consider combining providers." Your product can use OpenAI for complex reasoning + Modal for cheap high volume + Ollama local for dev. The choice doesn't have to be one-size-fits-all.
Exercise
Adapt this table to your product. Specifically:
- Remove dimensions you don't care about. If you don't do multimodal, delete the row. If you don't need HIPAA, delete that one.
- Add dimensions you do. Do you care about Spanish support? Latency from your specific region? Compatibility with your API gateway?
- Replace the generic numbers with benchmarks you measured in capsules 02-04.
- Mark your top 3 with a one-line explanation each.
- Paste it into your repo (
docs/llm-providers.md) or the team's Notion.
This is shared team infrastructure. When a PM asks "do we use OpenAI or Anthropic for X feature?", the table is the first answer.
Summary
You have:
- ✅ A master table structured by technical, economic, operational and compliance dimensions
- ✅ A use-case → primary recommendation + alternatives mapping
- ✅ A quick decision tree for a conversation with stakeholders
- ✅ Recommendations on how to use tables in technical discussions
Checkpoint: if you open this capsule 6 months later and can decide in 30 seconds which provider to evaluate for a new case, the table is doing its job.
Next capsule
08 — Project: decision tool. We close the module by building an automated Python tool that takes a requirements profile as input (volume, target latency, budget, constraints) and emits a recommendation with justification. It's the executable materialization of everything learned in M07.
Resources
- Artificial Analysis — Provider Comparison — updated public comparative benchmarks.
- LLM Performance Leaderboard (Vellum) — quality benchmarks.
- LMSys Chatbot Arena — paired human-eval comparison.
- Awesome LLM Ops — curated list of tools.