Module 7: Technical comparison of providers

You've done six modules. You know OpenAI, LM Studio, Ollama, OpenRouter and Modal. Each one works. If someone asks you "which one do I use for my project?", until now the answer was qualitative: "it depends on the case".

This module gives you the quantitative answer. You're going to run your own benchmarks (latency, cost, quality), build a data-based decision matrix, and end with a Python tool that automates the recommendation.

By the end of the module you'll be able to:

Measure P50, P95, P99 latency for each provider with a reproducible harness
Calculate the real cost per million requests for your specific load
Evaluate quality using a set of representative prompts and defensible metrics
Compare in a multi-criteria matrix and explain the trade-off of each choice
Migrate between providers with minimal code changes
Automate the decision with a decision tool based on requirements

Where are we in the path?

Module	Status
1-6	You learned how to use each provider
7	You learn how to decide with data ← you are here
8	You build the unified client that abstracts providers

After M07 you'll have something that very few AI engineers at your level can do: defend a provider choice with your own benchmarks in a technical meeting.

Why this module is important professionally

Three concrete reasons:

1. A differentiator in interviews. "I chose OpenAI because it's the most well-known" vs "I chose OpenAI because my benchmark showed a P95 of 1.4s vs 4.2s for local Ollama, my budget covers $X/month, and my model has no data restrictions". The second is hireable; the first is noise.

2. It reduces costly mistakes. Teams pay thousands of dollars on badly sized infrastructure because they chose "the most obvious" without measuring. One hour benchmarking saves weeks of migration later.

3. It gives you technical authority. When your cofounder asks "why Modal and not AWS?", the difference between fumbling and showing numbers is the difference between them trusting you or not.

A scenario that illustrates the module

Imagine you're the AI lead at a B2B support startup. Your CTO tells you:

"We want to offer a chatbot that answers technical questions about our product. We expect 500K queries/month. Some clients are enterprise and contract us with privacy clauses. We have $2k/month of initial budget. We need responses in <5 seconds. What stack do you recommend?"

Without this module, your answer is a hunch. With this module, your answer is:

I benchmark OpenAI GPT-4o-mini, OpenRouter (Mixtral), Modal with Mistral 7B → measured P95 latencies
I calculate cost-per-request on my expected traffic → 3 monthly scenarios
I test 30 representative technical support prompts → I compare quality
Multi-criteria table: latency × cost × quality × privacy
Final recommendation with numbers, not opinions

That's what you'll be able to do at the end of the module.

Module map

Capsule	Topic	What you build
01	Introduction (this one)	Mental model + objectives
02	Latency benchmark	Python harness that measures P50/P95/P99
03	Cost benchmark	Cost-per-request calculator with multiple scenarios
04	Quality benchmark	Evaluation set + manual scoring or LLM-as-judge
05	Advanced decision matrix	Weighted multi-criteria, not a "table with little stars"
06	Migration paths	How to switch providers with minimal changes
07	Full comparison table	The consolidated view: one table with everything
08	Project: decision tool	Python function that recommends given a requirements profile

Connection with the path's final project

What you learn here feeds Module 8 (Unified Client):

Your benchmarks tell you which provider to set as the default of the unified client
Your cost decision tells you what fallback order to use (cheapest first, fallback to the expensive one)
Your decision matrix gives you the mental model that the unified client encapsulates

Without M07, M08 is code without justification. With M07, M08 is the materialization of your analysis.

What is NOT covered

To keep the scope:

❌ Benchmarks of fine-tuned models. We assume you use base models. If you have fine-tunes, add them to the benchmark but the method is the same.
❌ Distributed benchmarks (load testing with thousands of clients). We see "single client measures latency" type benchmarks. For real load testing you need Locust, k6 or similar — out of scope.
❌ Scientifically rigorous quality comparison. We do LLM-as-judge and manual evaluation; for academic papers you need multiple human evaluators, inter-rater kappa, etc.
❌ Benchmarks of providers not covered (Anthropic Claude, Google Gemini, Together.ai, Replicate). The method transfers — add them yourself if your case requires it.

Common traps while taking the module

Trap 1 — "I measure once and declare a winner." Latency varies a lot. Measure 50-100 requests per provider to have defensible percentiles. A sample of 1 is noise.

Trap 2 — "I compare prices without understanding tokenization." "$0.50 per 1M tokens" means different things depending on whether your average request uses 100 or 2000 tokens. Always calculate cost per request in your case, not nominal price.

Trap 3 — "I compare different models and declare absolute winners." GPT-4o vs Mistral 7B isn't a fair comparison of provider — it's a comparison of model. Be explicit about what you're comparing: the provider (infrastructure) or the model (capability).

Trap 4 — "Choosing by benchmark alone." The numbers are an input, not the complete decision. Legal restrictions, enterprise support, team dependencies — everything plays in. Capsule 05 shows you how to combine them.

Self-assessment question

Before moving on to 02:

What's the difference between average latency and P95 latency? Why does P95 matter for a real product?
If OpenAI costs $5/1M tokens and Mistral on Modal costs $0.30/1M tokens, which is cheaper for 100K requests with 800 tokens on average? (calculate)
Why is "quality" the hardest of the three dimensions (latency, cost, quality) to measure?

Answer guide

Average is affected by outliers — a 30s request drags the mean up but "averages out" with many fast ones. P95 says "95% of my users wait less than this"; much more faithful to the real experience. If your P50 is 2s but your P95 is 25s, 5% of users wait 25s — for many products that's unacceptable even if the average looks fine.
OpenAI: 100,000 × 800 × $5 / 1,000,000 = $400. Mistral on Modal: 100,000 × 800 × $0.30 / 1,000,000 = $24. Mistral 16× cheaper in this scenario. (Assuming prices per 1M tokens; check the current prices of each provider.)
Because latency and cost are objective (numbers measured directly). Quality is subjective (what is "a better response"?). Capsule 04 covers how to make it as objective as possible: rubrics, LLM-as-judge, evaluation against ground truth.

Evidence of success at the end of the module

You'll know you finished well if you can:

✅ Run python benchmark.py and get a table with P50/P95/P99 for each provider
✅ Show the estimated monthly cost of each option for 3 different traffic scenarios
✅ Justify which provider you'd recommend for a specific case with numbers
✅ Have a reusable decision tool that you take to other projects

Next capsule

02 — Latency benchmark kicks off the quantitative work. You're going to build a Python harness that:

Measures P50/P95/P99 for each provider with a set of common prompts
Warms up the providers before measuring (avoids contaminating with cold starts)
Exports results to JSON/CSV for later analysis

It's the first of three measurements (latency, cost, quality) that you'll later combine in the decision matrix.

Resources for the module

Artificial Analysis Benchmarks — public benchmarks of latency and quality of LLM providers (a reference, not the source of truth for your case).
Anyscale — How to benchmark LLMs — technical benchmarking methodology.
HumanEval, MMLU, BBH — standard quality evaluation datasets.
LangSmith — LLM-as-judge — tool for automatic evaluation.
The art of choosing LLM providers (a16z) — market perspective.

← PreviousProject: production-ready scalable LLM API Next →Latency benchmark