Cost benchmark

Latency tells you whether your product feels good. Cost tells you whether your product is viable. An endpoint with a P50 of 1s but that costs $10,000/month is a different problem from one with a P50 of 4s but $200/month.

In this capsule you're going to build a calculator that estimates the real monthly cost of each provider for different traffic profiles. You're not going to use "the nominal price per 1M tokens" — you're going to calculate what you actually pay considering warm pool, storage, free tier, everything.

By the end you'll be able to:

• Calculate the approximate monthly cost for any provider given your volume and average request length
• Compare providers at 3 traffic scales (low/medium/high) and find the "cross-over point" where the winner changes
• Detect hidden costs that most people miss (warm pool, egress, model storage)
• Make a defensible spreadsheet to show a cofounder or finance

Why it matters

Three real scenarios where this changes decisions:

Scenario 1. An MVP startup estimates "we're going to use OpenAI to start". They do the math: $0.15 per 1M input tokens, seems cheap. They don't consider that their average request includes 3000 tokens of RAG context. The real cost is 20× what they estimated.

Scenario 2. A team decides on self-hosting with Ollama "because it's free". They don't include the cost of the VM with a GPU running 24/7, which ends up being $1,800/month. More expensive than OpenAI at their current volume.

Scenario 3. An agency uses Modal and it looks cheap. They don't notice that their min_containers=2 costs them $1,400/month of warm pool even though there's almost no nighttime traffic.

All of these are avoidable with 15 minutes of a spreadsheet. This capsule teaches you to do it right.

Mental model: cost types

Cross-over points: the line where one type stops being cheaper than another. Identifying them is the point of this capsule.

The base formula

For token-based providers:

monthly_cost = monthly_requests × avg_tokens_per_request × price_per_token

For GPU-based providers:

monthly_cost = (monthly_requests × seconds_per_request × price_per_second)
             + (warm_hours × 3600 × price_per_second)
             + model_storage_cost

For self-hosted (VM 24/7):

monthly_cost = monthly_hours × VM_price_per_hour
             + amortized_setup_cost
             + maintenance_cost (your time $$$)

Calculator: implementation

Create cost_calculator.py:

# cost_calculator.py
from dataclasses import dataclass
from typing import Literal

# ============================================================
# Reference prices (verify the current ones in each provider's docs)
# These are approximate as of early 2026; check before using.
# ============================================================
PRICES = {
    "openai_gpt-4o-mini": {
        "input_per_1m_tokens": 0.15,
        "output_per_1m_tokens": 0.60,
    },
    "openai_gpt-4o": {
        "input_per_1m_tokens": 2.50,
        "output_per_1m_tokens": 10.00,
    },
    "anthropic_claude-3.5-sonnet": {
        "input_per_1m_tokens": 3.00,
        "output_per_1m_tokens": 15.00,
    },
    "openrouter_mistral-7b-instruct": {
        "input_per_1m_tokens": 0.07,
        "output_per_1m_tokens": 0.07,
    },
    "openrouter_mixtral-8x7b-instruct": {
        "input_per_1m_tokens": 0.24,
        "output_per_1m_tokens": 0.24,
    },
    "modal_a10g": {
        "per_second": 0.000306,    # GPU only
    },
    "modal_t4": {
        "per_second": 0.000164,
    },
    "modal_a100_40gb": {
        "per_second": 0.001097,
    },
    "selfhosted_a10g_aws": {
        "per_hour": 1.006,         # g5.xlarge spot approx
    },
    "selfhosted_a100_aws": {
        "per_hour": 4.10,          # p4d.24xlarge spot approx
    },
}


# ============================================================
# Usage profile
# ============================================================
@dataclass
class UsageProfile:
    requests_month: int
    avg_input_tokens: int = 500
    avg_output_tokens: int = 200
    avg_gpu_seconds: float = 2.0   # for GPU providers
    warm_pool_hours_month: float = 0      # 0 if min_containers=0; 720 if 24/7

    def cost_openai(self, model: str) -> float:
        p = PRICES[f"openai_{model}"]
        return (
            self.requests_month * self.avg_input_tokens * p["input_per_1m_tokens"] / 1_000_000
            + self.requests_month * self.avg_output_tokens * p["output_per_1m_tokens"] / 1_000_000
        )

    def cost_anthropic(self, model: str) -> float:
        p = PRICES[f"anthropic_{model}"]
        return (
            self.requests_month * self.avg_input_tokens * p["input_per_1m_tokens"] / 1_000_000
            + self.requests_month * self.avg_output_tokens * p["output_per_1m_tokens"] / 1_000_000
        )

    def cost_openrouter(self, model: str) -> float:
        p = PRICES[f"openrouter_{model}"]
        total_tokens = self.avg_input_tokens + self.avg_output_tokens
        return self.requests_month * total_tokens * p["input_per_1m_tokens"] / 1_000_000

    def cost_modal(self, gpu: str = "a10g") -> float:
        price_sec = PRICES[f"modal_{gpu}"]["per_second"]
        active_gpu = self.requests_month * self.avg_gpu_seconds * price_sec
        warm_pool = self.warm_pool_hours_month * 3600 * price_sec
        storage_model = 2.0  # ~14GB × $0.10/GB/month ≈ $1.40, rounded
        return active_gpu + warm_pool + storage_model

    def cost_selfhosted(self, gpu: str = "a10g_aws") -> float:
        per_hour = PRICES[f"selfhosted_{gpu}"]["per_hour"]
        return 720 * per_hour  # 24×30 = 720 hrs/month

    def cost_ollama_local(self) -> float:
        return 0.0  # your hardware/electricity, not billed directly


# ============================================================
# Comparative report
# ============================================================
def report(profile: UsageProfile, label: str):
    print(f"\n=== {label} ===")
    print(f"  Volume: {profile.requests_month:,} requests/month")
    print(f"  Tokens: {profile.avg_input_tokens} input + {profile.avg_output_tokens} output")
    print(f"  Warm pool: {profile.warm_pool_hours_month} hrs/month\n")

    print(f"  {'Option':<45} {'Cost/month':>12}")
    print(f"  {'-' * 60}")

    candidates = [
        ("OpenAI GPT-4o-mini", profile.cost_openai("gpt-4o-mini")),
        ("OpenAI GPT-4o", profile.cost_openai("gpt-4o")),
        ("Anthropic Claude 3.5 Sonnet", profile.cost_anthropic("claude-3.5-sonnet")),
        ("OpenRouter Mistral 7B", profile.cost_openrouter("mistral-7b-instruct")),
        ("OpenRouter Mixtral 8x7B", profile.cost_openrouter("mixtral-8x7b-instruct")),
        ("Modal Mistral 7B (A10G)", profile.cost_modal("a10g")),
        ("Modal Mistral 7B (T4)", profile.cost_modal("t4")),
        ("Self-hosted Ollama (AWS A10G)", profile.cost_selfhosted("a10g_aws")),
        ("Ollama local (your hardware)", profile.cost_ollama_local()),
    ]

    for name, cost in sorted(candidates, key=lambda x: x[1]):
        marker = " ←" if cost == min(c for _, c in candidates) else ""
        print(f"  {name:<45} ${cost:>10.2f}{marker}")


# ============================================================
# Three representative profiles
# ============================================================
if __name__ == "__main__":
    LOW = UsageProfile(
        requests_month=10_000,
        avg_input_tokens=300,
        avg_output_tokens=150,
        avg_gpu_seconds=1.5,
        warm_pool_hours_month=0,
    )

    MEDIUM = UsageProfile(
        requests_month=200_000,
        avg_input_tokens=500,
        avg_output_tokens=200,
        avg_gpu_seconds=2.0,
        warm_pool_hours_month=200,  # warm pool during business hours
    )

    HIGH = UsageProfile(
        requests_month=2_000_000,
        avg_input_tokens=800,
        avg_output_tokens=300,
        avg_gpu_seconds=2.5,
        warm_pool_hours_month=720,  # warm pool 24/7
    )

    report(LOW, "LOW: MVP / hobby (10k req/month)")
    report(MEDIUM, "MEDIUM: Product in growth (200k req/month)")
    report(HIGH, "HIGH: Established product (2M req/month)")

Run:

python cost_calculator.py

Typical reading of results

Approximate outputs (with 2026 numbers, adjust to current prices):

=== LOW: MVP / hobby (10k req/month) ===
  Ollama local (your hardware)                   $      0.00 ←
  OpenAI GPT-4o-mini                             $      1.65
  OpenRouter Mistral 7B                          $      0.32
  Modal Mistral 7B (T4)                          $      4.46
  OpenAI GPT-4o                                  $     27.50
  ...
  Self-hosted Ollama (AWS A10G)                  $    724.32

=== MEDIUM: Product in growth (200k req/month) ===
  OpenRouter Mistral 7B                          $      9.80 ←
  Ollama local                                   $      0.00 (excluded — doesn't scale this way)
  OpenAI GPT-4o-mini                             $     39.00
  Modal Mistral 7B (T4) + warm pool              $    187.20
  ...
  OpenAI GPT-4o                                  $    900.00
  Self-hosted Ollama (AWS A10G)                  $    724.32

=== HIGH: Established product (2M req/month) ===
  Self-hosted Ollama (AWS A10G)                  $    724.32 ←
  Modal Mistral 7B (T4) full warm                $  1,200.00 (approx)
  OpenRouter Mistral 7B                          $    154.00
  OpenAI GPT-4o-mini                             $    660.00
  OpenAI GPT-4o                                  $ 12,000.00

Patterns that emerge:

• Low: local "wins" if your hardware is enough. OpenRouter Mistral is scandalously cheap if you don't need GPT-4 quality.
• Medium: OpenRouter still wins for cases where Mistral is enough. OpenAI/Anthropic win if your quality requires it.
• High: self-hosted starts to make sense. Paying for a GPU 24/7 vs paying per usage changes the math.

Cross-over points: where the winner changes

Interesting question: at what volume does Self-hosted beat OpenAI?

# Solve: 720 × $1 = req × 700 × ($0.15 + $0.60) / 1M (gpt-4o-mini)
# $720 = req × 525 / 1M
# req = 1.37M requests/month

With 700 total tokens/request, OpenAI GPT-4o-mini is cheaper than Self-hosted A10G up to ~1.4M req/month. Past that point, self-hosted wins.

But — and this is important — Self-hosted assumes 100% utilization (24/7 traffic saturating the GPU). If your traffic is bursty (200K during business hours, 0 at dawn), the GPU is idle 50% of the time and you need to multiply your break-even by ~2×.

Lesson: cross-over points depend on your traffic pattern, not just the total volume.

Hidden costs that almost nobody includes

1. Warm pool. Modal with min_containers=1 A10G = ~$720/month extra. Easy to forget; painful at the end of the month.

2. Egress. Modal and cloud providers charge for outbound transfer. For text chat it's negligible; for images/audio it can hurt.

3. Model storage. 50GB volumes with several models = ~$5/month. Cumulative.

4. Human maintenance (self-hosted). If your team spends 5hrs/week maintaining the GPU, patching drivers, debugging OOM — that's engineer time. At $80/hr × 20hrs/month = $1,600/month "invisible".

5. Logging and observability. Datadog, Sentry, etc. add ~$50-300/month. You need it but it doesn't show up in the LLM provider calc.

6. Tier creep in OpenAI. "We're going to start with gpt-4o-mini" → "we need better quality, we move to gpt-4o" → a bill 15× higher. Plan for that drift.

Common traps

Trap 1 — "OpenAI says $0.15 per 1M tokens, so cheap." That's for input. Output costs 4×. And your real request adds input + output. Read the full prices.

Trap 2 — "Ollama is free." Your hardware isn't free ($2000 laptop, ~$15/month electricity running full-time). For personal use, "free enough". For a commercial product, consider amortization.

Trap 3 — "Self-hosted is always cheaper at scale." Only if the GPU is saturated. If your nighttime traffic is 5% of daytime, a GPU running 24/7 wastes 50%+ of the time. Modal/serverless can still win.

Trap 4 — "I compare Mistral 7B vs GPT-4 by price." It's not a fair comparison. If Mistral 7B doesn't solve your task with acceptable quality, its "low price" is irrelevant. Compare models that meet the quality SLA, not just any model.

Trap 5 — "I calculated with prices from a year ago." Prices change. OpenAI lowered gpt-4o-mini significantly in 2025. Anthropic changed pricing. Modal adjusts. Check current prices when you're going to make a decision.

Exercise

Your product is a B2B technical support assistant:

• 80K requests/month expected
• Each request includes 1500 tokens of RAG context + 200 tokens of question = ~1700 input
• Average response ~250 tokens
• Traffic concentrated from 9am-6pm business hours, almost nothing at night
• Your biggest enterprise client demands that the data not leave the US (cloud-managed providers get a ⚠️)

Calculate:

1. Approximate monthly cost with OpenAI GPT-4o-mini (is it valid for your enterprise client?)
2. Approximate monthly cost with Modal Mistral 7B + warm pool during business hours
3. Approximate monthly cost with Self-hosted Ollama in the EU
4. Which do you recommend and why?

case_profile = UsageProfile(
    requests_month=80_000,
    avg_input_tokens=1700,
    avg_output_tokens=250,
    avg_gpu_seconds=2.5,
    warm_pool_hours_month=200,  # ~9hrs × 22 days
)

# OpenAI gpt-4o-mini
# = 80000 × 1700 × 0.15 / 1M + 80000 × 250 × 0.60 / 1M
# = $20.40 + $12.00 = $32.40
# ⚠️ NOT valid if the enterprise client restricts data residency in the US

# Modal Mistral 7B A10G
# Active GPU: 80000 × 2.5 × $0.000306 = $61.20
# Warm pool: 200 × 3600 × $0.000306 = $220.32
# Storage: ~$2
# Total: ~$283.52
# (verify Modal's SOC2/data residency certification with your client)

# Self-hosted Ollama EU
# A10G on AWS EU eq.: ~$1.05/hr × 720 = $756/month
# + human maintenance cost (variable)

# Recommendation:
# - If Modal meets your client's data residency (verify), Modal is ~3× cheaper
# - If not, Self-hosted EU is the only valid option and you accept the cost
# - OpenAI isn't viable due to the contractual restriction

Summary

You learned:

• ✅ Cost types (token, GPU-second, fixed, free) and when each one wins
• ✅ Calculate monthly cost including hidden costs (warm pool, storage, egress, human)
• ✅ Identify cross-over points where the economic winner changes
• ✅ Design representative profiles (low/medium/high) that reflect typical growth
• ✅ Defend a recommendation with numbers, not opinions

Checkpoint: if you can answer "how much does this cost a month?" without fumbling, with a number justified by a formula, you're ready.

Next capsule

04 — Quality benchmark. The hardest of the three dimensions. We're going to pin it down with: a representative evaluation set, a clear rubric, LLM-as-judge, and defensible metrics.

Resources

1. OpenAI Pricing — source of truth, changes.
2. Modal Pricing — current GPU per second.
3. OpenRouter Models — aggregated prices per model.
4. Anthropic Pricing — Claude variants.
5. Artificial Analysis — Pricing Comparisons — public comparative table.
6. AWS EC2 GPU Pricing — for self-hosted calculation.