Module 7: Technical comparison of providers

Migration paths

Next →

Your matrix from capsule 05 told you "provider X wins today". But today doesn't last forever:

  • OpenAI cuts a model's price in half → it's worth migrating
  • Your new client requires data residency in the EU → you have to leave the US
  • An open-source model comes out with quality comparable to GPT-4 → the math changes
  • Your volume goes up 10× and self-hosted starts to win

The question stays the same: how much work does it cost to switch? If the answer is "1 day", you migrate whenever it's convenient. If it's "3 refactor sprints", you're stuck with the provider even if it stops being optimal. That friction is called vendor lock-in, and reducing it is real value.

By the end you'll be able to:

  • Estimate the migration cost between any pair of providers in the path
  • Identify the APIs and abstractions that minimize lock-in
  • Design your code from the start so migrating is cheap
  • Execute a real migration with a concrete checklist

Why it matters

There are two ways to prepare for uncertainty:

  1. Guess the future and pick the "definitive winner" → almost always fails
  2. Reduce the cost of switching → always works

Method 2 is the only robust strategy for a market that changes every quarter. This capsule is about method 2.

Mental model: the hierarchy of APIs

Good news: the ecosystem converged. Most providers adopted (formally or de facto) the OpenAI format as the standard interface. That means many migrations are a URL change + API key change, not a rewrite.

                              ┌─→ OpenAI (native)
   OpenAI Chat format         ├─→ OpenRouter (official OpenAI-compatible proxy)
   /v1/chat/completions  ──→  ├─→ Ollama (OpenAI-compatible mode)
                              ├─→ LM Studio (OpenAI-compatible server)
                              ├─→ Modal (your custom API, but you can copy the format)
                              └─→ Together, Anyscale, Groq, vLLM, etc.

Anthropic Claude and Google Gemini have their own formats but also offer OpenAI-compatible endpoints (through official proxies or an adaptation layer).

Migration 1 — OpenAI → OpenRouter

Effort: ~5 minutes. A 2-line change.

# BEFORE
from openai import OpenAI
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": prompt}],
)
# AFTER — OpenRouter
from openai import OpenAI
client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
)

response = client.chat.completions.create(
    model="mistralai/mistral-7b-instruct",  # ← pick any model from the catalog
    messages=[{"role": "user", "content": prompt}],
)

What changes: base_url, api_key, model.

What does NOT change: messages, temperature, max_tokens, response_format, streaming handling, error handling.

Post-migration verification:

  • Existing tests pass
  • Latency, quality and cost behave as your benchmark predicted

Migration 2 — OpenAI → Ollama (local)

Effort: ~10 minutes if Ollama is already running. A 3-line change.

# AFTER — Ollama local
from openai import OpenAI
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # any string; Ollama ignores it
)

response = client.chat.completions.create(
    model="mistral",  # your Ollama model
    messages=[{"role": "user", "content": prompt}],
)

What changes: base_url, api_key (placeholder), model.

Catch: some OpenAI parameters aren't supported by Ollama (response_format with advanced JSON Schema, full function calling with multi-turn, etc). Verify the ones you use.

Migration 3 — OpenAI → LM Studio

Effort: identical to Ollama. LM Studio exposes an OpenAI-compatible endpoint at http://localhost:1234/v1.

client = OpenAI(
    base_url="http://localhost:1234/v1",
    api_key="lm-studio",
)

Same pattern. The difference is operational (model loaded in the LM Studio GUI vs the Ollama CLI).

Migration 4 — Cloud (OpenAI/OpenRouter) → Modal

Effort: ~1-2 hours. It's not a URL change — it's a new deployment.

Steps:

  1. Define the Modal function (decorators, image, GPU) — copy from Module 6
  2. Deploy (modal deploy)
  3. Your endpoint exposes /chat with your custom contract (not native OpenAI)
  4. Adapt the caller to use your endpoint
# The caller that used to hit OpenAI directly
import httpx

def chat_via_modal(prompt: str) -> str:
    r = httpx.post(
        os.environ["MODAL_BASE_URL"] + "/chat",
        headers={"Authorization": f"Bearer {os.environ['MODAL_TOKEN']}"},
        json={"prompt": prompt, "max_tokens": 256},
        timeout=60,
    )
    r.raise_for_status()
    return r.json()["response"]

If you want to keep OpenAI compatibility: design your Modal endpoint to emit the chat.completions.create() response format and expose it at /v1/chat/completions. Then your Modal becomes a drop-in replacement for OpenAI (which is what you'll do conceptually in Module 8).

Migration 5 — Modal → Self-hosted (Ollama on AWS)

Effort: ~1 week. It's the biggest change in the module.

Requires:

  1. Provision the VM with a GPU (AWS g5.xlarge, Lambda Cloud, etc.)
  2. Install CUDA drivers + Ollama (Docker simplifies it)
  3. Download the model (ollama pull mistral)
  4. Configure a reverse proxy with HTTPS (Caddy, Nginx + Let's Encrypt)
  5. Set up observability (logs, metrics, alerts)
  6. Adapt your client to the new URL

Reason to do it: economics at high volume, compliance, total control. Reason not to: the team doesn't want to maintain infrastructure.

Migration 6 — OpenAI → Anthropic Claude

Effort: ~30 minutes. A different format.

Anthropic has its own API (messages.create() instead of chat.completions.create()). Key differences:

  • system goes as a top-level parameter, not as a message
  • Streaming and tool use have different formats
  • Separate pricing for input/output cache

If your code only does a simple chat.completions.create(), there's a trivial wrapper:

import anthropic
client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])

def chat(prompt: str, system: str = "") -> str:
    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=400,
        system=system or "You are a helpful assistant.",
        messages=[{"role": "user", "content": prompt}],
    )
    return response.content[0].text

Alternative: use Anthropic via OpenRouter or via an OpenAI-compatible proxy → you're back to the base_url + model pattern.

The pattern that minimizes migrating: an abstraction layer

What Module 8 (Unified Client) is going to build is exactly this:

class UnifiedClient:
    def __init__(self, provider: str, **kwargs):
        self.provider = provider
        self._client = self._build_client(provider, **kwargs)

    def chat(self, prompt: str, **opts) -> str:
        """Single interface; the per-provider adapter translates."""
        return self._client.chat(prompt, **opts)

Your application code never touches OpenAI directly, Modal directly, etc. It only touches UnifiedClient. Migrating is a change of one parameter:

# Before
client = UnifiedClient(provider="openai")

# After
client = UnifiedClient(provider="modal")

That abstraction has a cost: you restrict yourself to the intersection of the providers' features. If you need sophisticated function calling that only OpenAI supports well, the abstraction becomes awkward. It's a trade-off — but for many products where advanced features don't apply, it's worth it.

Migration checklist (any direction)

When you execute a real migration:

  • Baseline benchmark. Before migrating, measure latency/cost/quality of the current provider with your real set. Without this, you won't know whether the migration improved anything.
  • Regression tests. Your evaluation set from capsule 04. Run it before and after. Significant differences deserve an explanation.
  • Side-by-side comparison in production. Send 10% of the traffic to the new provider, compare latency and errors. Modal/OpenRouter/Anthropic all tolerate this pattern with feature flags.
  • Rollback plan. What happens if the new provider fails? A clear plan to return to OpenAI / the previous one. Keep the old code for 1-2 sprints.
  • Update the projected cost. Your finance dashboard / cofounder should see the change in costs.
  • Update the docs. README, runbooks, alerts. The step everyone forgets.
  • Update client contracts (if applicable). A provider change may trigger notification clauses (especially data processing addendums).

Migration costs by scenario

MigrationEng effortRiskWhen it's worth it
OpenAI ↔ OpenRouter30 minLowAlmost always, especially for an A/B test
OpenAI ↔ Ollama local1 hourLow (dev environments)Dev / privacy / costs
OpenAI ↔ Anthropic1-2 hrsMedium (different format)Your quality improves >10%
Cloud → Modal1-2 daysMediumBurst volume + model control
Cloud → Self-hosted1-2 weeksHighVery high, constant volume + compliance

Common traps

Trap 1 — "I switched providers but forgot the prompt template." Each model has its ideal template. Mistral uses [INST] ... [/INST]. Llama 3 uses another. ChatML for many. If you copy the same prompt without formatting, quality drops for no apparent reason. Check the model cards.

Trap 2 — "I got stuck on an exclusive feature." If you use OpenAI's sophisticated function calling with multiple concurrent tools, migrating to an open-source model that doesn't support it requires a major rewrite. Design assuming you only have the common set and add advanced features as optional improvements.

Trap 3 — "I migrated and now I have two providers running." It happens: you wanted to replace OpenAI with OpenRouter, but you left the old code "just in case" and never deleted it. Result: two integrations to maintain, two sets of keys, double billing. Delete the old code as soon as the migration is stable.

Trap 4 — "My rollback plan was 'go back to the previous branch'." If your new code persisted data in a different format, "going back" isn't trivial. Consider a feature flag that lets you switch without a redeploy.

Trap 5 — "I underestimated the pricing change." Technically successful migration, unexpected bill. Re-benchmark cost with real data in the first week post-migration, don't wait for the end-of-month close.

Exercise

Your current product uses OpenAI GPT-4o-mini. Your CTO asks you to prepare 3 migration playbooks for different scenarios:

  1. Playbook A: OpenAI cuts gpt-4o-mini by 50%. Do you change anything? Do you re-benchmark?
  2. Playbook B: Your biggest client demands EU data residency. You have 30 days.
  3. Playbook C: Your volume quintuples from 100K req/month to 500K req/month in a quarter.

For each one: effort estimate, risks, recommended decision (with a reason).

See answer guide

Playbook A — gpt-4o-mini price cut:

  • Effort: 0 (you're already there)
  • Action: re-benchmark projected cost to validate that your bill drops as you expect; update the decision matrix in case the change affects competitors; communicate the savings to the team.
  • Decision: stay, monitor whether Anthropic / OpenRouter / Modal make a counter-move.

Playbook B — EU data residency in 30 days:

  • Viable options: OpenAI EU (if certified), Anthropic EU, OpenRouter with an EU filter, Modal in an EU region (verify), Self-hosted EU.
  • Effort: 1-2 weeks (depends on the option)
  • Risks: quality re-benchmark (EU models may have variants), latency from your servers, possible quality differences if the model changes
  • Recommended decision: start an A/B test immediately with the most likely EU option; in parallel, validate legal/compliance; be ready to deploy on day 25 with a rollback plan.

Playbook C — 5× volume:

  • Current projected cost: $200/month → $1000/month (gpt-4o-mini)
  • Cross-over point with Self-hosted: it probably still doesn't pay off (Self-hosted = ~$720/month with A10G 24/7, assuming 70% utilization)
  • Main risk: rate limits of the current OpenAI tier
  • Effort if you migrate to Self-hosted: 1-2 weeks + ongoing maintenance
  • Recommended decision: keep OpenAI; raise the tier; add caching of frequent responses (-30% of requests easily); re-evaluate in 6 months with real data.

Summary

You learned:

  • ✅ The OpenAI format is the de facto standard — most migrations are a URL change
  • ✅ Five key migrations: OpenAI↔OpenRouter (5min), OpenAI↔Ollama (10min), Cloud↔Modal (hrs), Modal↔Self-hosted (days), OpenAI↔Anthropic (30min)
  • ✅ The abstraction layer (Unified Client) reduces migration to a parameter change
  • ✅ Migration checklist: baseline, tests, side-by-side, rollback, docs
  • ✅ Traps: prompt template, exclusive feature, two providers coexisting, fragile rollback

Checkpoint: if you can estimate in 30 seconds "how much does it cost to switch from X to Y?" for any pair in the path, you're ready.

Next capsule

07 — The full comparison table. We consolidate everything learned into a single table that serves as a quick reference for your team. It's the capsule you'll bookmark and re-read whenever you have to decide.

Resources

  1. OpenAI API reference — the de facto standard.
  2. Ollama OpenAI compatibility — Ollama's OpenAI-compatible mode.
  3. Anthropic — Comparing to OpenAI — official migration guide.
  4. LiteLLM — an "OpenAI client for any provider" type library, inspiration for Module 8.
  5. Architectural Decision Records — pattern for documenting migrations.
← PreviousAdvanced decision matrixNext →The full comparison table