Module 7: Performance vs Cost Trade-offs

Model routing and multi-model

Next →

Overview

GPT-4o costs $0.0025 per 1K input tokens. Mistral 7B on Modal costs ~$0.0001 for the equivalent. A 25× difference. The obvious question: can you use the cheap model for most queries and the expensive one only when it matters?

Yes, and it's called model routing. It's one of the highest-leverage trade-offs you have: it can typically reduce your LLM cost 40-70% without sacrificing quality where it matters.

In this lesson you'll learn the framework for intelligent routing: when you use which model, how to automatically classify the input, and how to measure whether your routing is actually working.

By the end you'll be able to:

  • Identify which queries can go to cheaper models
  • Implement routing rules (manual or ML-based)
  • Design ensemble patterns (a cheap model as a first pass, escalate to an expensive one)
  • Measure effectiveness: cost reduction vs quality degradation

The mental model

Three typical tiers:

Tier 3: GPT-4o, Claude 3 Opus
   ↑    $0.01-$0.05 per request
   ↑    For: complex reasoning, code, math, critical queries
   │
Tier 2: GPT-4o-mini, Claude Haiku, Mistral Large
   │    $0.0005-$0.002 per request
   │    For: standard queries, summarization, classification
   │
Tier 1: Mistral 7B, Llama 8B (self-hosted or OpenRouter)
   ↓    $0.00005-$0.0002 per request
   ↓    For: simple queries, factual lookup, structured formatting

Goal: route each query to the minimum tier that gives acceptable quality. Many people use Tier 3 for everything "to be safe", when 70% of queries do fine with Tier 1-2.

Routing strategies

Strategy 1: Explicit rule-based

The simplest. Your code classifies the query with rules:

def route_query(query: str, user: dict) -> str:
    """Returns model name to use."""

    # Tier 3 (expensive but better)
    if any(kw in query.lower() for kw in ["código", "code", "function", "debug"]):
        return "gpt-4o"
    if any(kw in query.lower() for kw in ["analiz", "compar", "razon"]):
        return "gpt-4o"
    if user["tier"] == "enterprise":
        return "gpt-4o"  # paying customers get the best

    # Tier 1 (cheap)
    if len(query.split()) < 10 and "?" in query:
        return "mistral-7b"  # short lookup-type query

    # Tier 2 (balanced default)
    return "gpt-4o-mini"

Pros: explicit, debuggable. Cons: rules can be miscalibrated; manual maintenance.

Strategy 2: ML Classifier

A small model classifies the query into one of N categories (e.g., "simple", "medium", "complex"). You route by category.

async def route_query_ml(query: str) -> str:
    # Small and fast classifier
    category = await classifier_model.predict(query)  # "simple" / "medium" / "complex"

    return {
        "simple": "mistral-7b",
        "medium": "gpt-4o-mini",
        "complex": "gpt-4o",
    }[category]

Pros: adapts to your data, better accuracy than rules. Cons: requires training and maintaining the classifier; extra latency from the classification call.

Strategy 3: Cascade (cascading fallback)

You start with the cheap model. If the result doesn't meet a quality threshold, you escalate to the expensive one.

async def route_cascade(query: str) -> str:
    # Tier 1 first
    response_cheap = await call_llm(query, "mistral-7b")

    # Confidence check (whatever is relevant)
    if response_cheap.confidence > 0.85:
        return response_cheap.text

    # Escalate to tier 2
    response_better = await call_llm(query, "gpt-4o-mini")
    if response_better.confidence > 0.90:
        return response_better.text

    # Tier 3 as a last resort
    response_best = await call_llm(query, "gpt-4o")
    return response_best.text

Pros: optimal automatically: each query consumes only what it needs. Cons: higher latency (it can take 2-3 calls), complex.

Strategy 4: Ensemble

You call multiple models in parallel and combine (voting, weighted average).

async def route_ensemble(query: str) -> str:
    responses = await asyncio.gather(
        call_llm(query, "mistral-7b"),
        call_llm(query, "gpt-4o-mini"),
    )
    # Voting: if both agree, return. If not, escalate.
    if responses_agree(responses):
        return responses[0].text
    return await call_llm(query, "gpt-4o")

Pros: high accuracy. Cons: pays for multiple models. Only worth it on critical queries where accuracy >> cost.

Patterns by product type

Product 1: General customer support

  • 70% queries: factual lookup ("how do I change my password?")
  • 25% queries: reasoning ("explain to me why X")
  • 5% queries: complex (multi-step, integration debugging)

Strategy: rule-based + cascade fallback.

Default: Tier 1 (cheap, fast)
If confidence is low: escalate to Tier 2
For queries with keywords "explain", "why", "compare": Tier 2
For queries of >100 words or with code: Tier 3

Expected cost reduction: ~60%.

Product 2: AI assistant for engineers

  • 50% queries: code generation
  • 30% queries: concept explanation
  • 20% queries: multi-step debugging

Strategy: an ML classifier detects the query type.

Code generation → GPT-4o (quality critical)
Explanation → GPT-4o-mini
Debugging → GPT-4o-mini with cascade to GPT-4o if confidence is low

Cost reduction: ~30% (expensive models still dominate in this vertical).

Product 3: Generalist B2C chatbot

  • 80% queries: conversational (chitchat, opinions)
  • 15% queries: factual lookups
  • 5% queries: specific tasks

Strategy: mainly Mistral 7B (cheap), escalate to GPT-4o-mini for fact-checks.

Default: Mistral 7B
If the query includes dates, numbers, specific facts: GPT-4o-mini (more reliable)

Cost reduction: ~75-85% (most queries are cheap chitchat).

How to measure whether the routing works

Critical metrics:

1. Cost reduction

Cost before: $X/month (all GPT-4o-mini)
Cost after: $Y/month (with routing)
Reduction: (X - Y) / X

2. Quality degradation

Before implementing: run your eval set (see M7 lesson 04 of Module 4 — Quality benchmark) against GPT-4o-mini. After routing: run the same eval set against your system with routing.

Quality before: 87% pass rate
Quality after: 84% pass rate
Degradation: 3 percentage points

It's acceptable if: degradation <5pp and cost reduction >30%.

3. Distribution per model

SELECT model_used, COUNT(*) as count
FROM llm_calls
WHERE timestamp > NOW() - INTERVAL '7 days'
GROUP BY model_used;

Example:

mistral-7b      | 145,000 (58%)
gpt-4o-mini     |  82,000 (33%)
gpt-4o          |  23,000 (9%)

If all queries go to Tier 3, your routing isn't working. If all go to Tier 1, quality is probably bad.

4. Misclassification rate (if you use an ML classifier)

% of queries the classifier routes wrong (manual validation).

Integrated Decision Matrix

Combining everything from M7-02 to M7-06:

For each component of the system:

1. Scale up vs out (M7-02):
   - Decision: ___
   - Justification: ___

2. Cache strategy (M7-03):
   - What to cache: ___
   - TTL: ___
   - Invalidation: ___

3. Managed vs self-hosted (M7-04):
   - Decision: ___
   - Calculated TCO: ___

4. Serverless vs containers (M7-05):
   - Decision: ___
   - Justification: ___

5. Model routing (M7-06):
   - Tier breakdown: __% / __% / __%
   - Cost estimate: ___
   - Quality target: ___

This is what we're going to formalize in the next lesson (Decision Matrix template).

Common traps

Trap 1 — Routing everything to tier 1. Super low cost, but terrible quality. Measure quality.

Trap 2 — Routing everything to tier 3 "to be safe". Cost disproportionate vs value. Accept that tier 1-2 covers 70%+ of queries.

Trap 3 — Not measuring misclassification. When your classifier routes wrong, complex queries go to tier 1 and give poor responses. Tracking "user marked as unhelpful" per tier exposes this.

Trap 4 — Ignoring the ML classifier's latency. If the classifier takes 200ms, and tier 1 takes 800ms, the total is 1s. vs calling tier 2 directly is 1.2s. You barely gain in latency, only in cost.

Trap 5 — A cascade that almost always escalates. If your tier 1 fails the quality threshold 90% of the time, you're paying tier 1 + tier 2 + tier 3. Worse than just tier 3.

Trap 6 — Hardcoded rules without updating. Your product evolves, queries change. The rules from 6 months ago aren't current. Audit rules quarterly.

Exercise

Your current system: you spend $4K/month on GPT-4o-mini (everything goes to the same model). 1M queries/month. Design a routing strategy.

Your data:

  • 60% of queries are factual lookups from the knowledge base (RAG)
  • 25% are concept explanations
  • 10% are more complex tasks (multi-step reasoning)
  • 5% are code

Specify:

  1. Which models in your cascade?
  2. Routing rules
  3. Expected cost reduction
  4. How you measure quality degradation
See solution

  1. Models:

    • Tier 1: Mistral 7B via OpenRouter ($0.07 + $0.07 / 1M tokens, ~$0.0001 per request)
    • Tier 2: GPT-4o-mini ($0.15 + $0.60 / 1M tokens, ~$0.001 per request)
    • Tier 3: GPT-4o ($2.50 + $10 / 1M tokens, ~$0.015 per request)

  2. Routing rules:

    if "código" in query OR ```` in query: → Tier 3
elif len(query) > 200 words: → Tier 3
elif any(["razón", "analiza", "compara", "explica el porqué"]): → Tier 2
elif "explica" in query (general): → Tier 2
else: → Tier 1 (RAG lookups)

  3. Estimated cost reduction:

    • 60% lookups → Tier 1: 600K × $0.0001 = $60
    • 25% explanations → Tier 2: 250K × $0.001 = $250
    • 10% complex → Tier 2 (with cascade to 3 if confidence is low): 100K × $0.001 + ~20% escalate = $100 + $300 = $400
    • 5% code → Tier 3: 50K × $0.015 = $750
    • Total: ~$1,460/month
    • Reduction: $4000 → $1460 = 63%

  4. Quality measurement:

    • An eval set of 100 representative queries (a mix of the 4 types)
    • Run the eval against: (a) the system without routing (all gpt-4o-mini, baseline), (b) the system with routing
    • Metric: LLM-as-judge with a rubric (correctness, relevance)
    • Acceptable: <3pp degradation overall
    • Per type: lookups <5pp, explanations <3pp, complex <2pp, code <1pp
    • If degradation exceeds the thresholds in any category, adjust the rules (e.g., raise complexity to Tier 2)

Summary

You learned:

  • ✅ The tier mental model (1/2/3) with typical costs
  • ✅ 4 strategies: rule-based, ML classifier, cascade, ensemble
  • ✅ Patterns by product type (support, engineering, B2C chatbot)
  • ✅ Measurement: cost reduction, quality degradation, distribution per model
  • ✅ Traps: routing everything to the same tier, not measuring misclassification, hardcoded rules

Checkpoint: if you can design a routing strategy with a quantified cost-quality trade-off, you're ready.

Next lesson

07 — Decision Matrix template. We close M7 by building the reusable tool you'll apply to M8 and any future project: the Decision Matrix with weighted criteria, scores, and documented outcomes.

Resources

  1. Martian — LLM router — managed routing service.
  2. RouteLLM (LM-SYS) — research on routing.
  3. Anthropic — Multi-model strategies — patterns.
  4. vLLM continuous batching — to optimize tier 1.
  5. LangChain RouterChain — a ready-made implementation.
← PreviousServerless vs ContainersNext →Decision Matrix template