Module 7: Performance vs Cost Trade-offs

Managed vs Self-hosted

Next →

Overview

"Pinecone managed or ChromaDB self-hosted? OpenAI or your own Llama? Cloud Run or your own Kubernetes?" These are the decisions that most impact your TCO (total cost of ownership) and your iteration speed as a startup.

There's a bias in the industry: "managed is always better because you don't want to operate infrastructure". It's false. Managed has real trade-offs: vendor lock-in, a cost that scales ugly with volume, less control over performance, dependence on their SLA. For some cases, self-hosted wins — not by dogma, by math.

This lesson gives you the component-by-component framework.

By the end you'll be able to:

  • Calculate the real TCO (not just the list price) for managed vs self-hosted
  • Evaluate the human operational cost (the most underestimated metric)
  • Decide component by component: LLM, vector DB, hosting, monitoring
  • Recognize common hybrid patterns (a mix of managed + self-hosted)

The framework: 5 dimensions

DimensionManaged wins whenSelf-hosted wins when
Absolute costLow-medium volumeHigh and sustained volume
Human costSmall team, no DevOpsTeam with DevOps skills
ControlYou don't care about fine tuningsYou need specific optimization
Vendor lock-inIt doesn't worry youStrategic: independence
Time to productionIt matters, MVP/early stageYou have time to develop it well

Simple rule for startups: you start managed (everything), you migrate to self-hosted whatever you can justify with numbers.

Component-by-component

LLM provider

Managed (OpenAI, Anthropic, OpenRouter):

  • Cost: $0.15-$15 per 1M tokens depending on the model
  • Setup: 5 minutes
  • Scaling: automatic
  • Vendor lock-in: medium (you can migrate but it requires testing)

Self-hosted (Llama, Mistral on your GPU):

  • Cost: ~$1.5K-$5K/month in GPU (A10G to H100)
  • Setup: 1-2 days for a functional deployment
  • Scaling: your problem
  • Vendor lock-in: zero

Typical cross-over: managed wins up to ~10M tokens/day. Beyond that, self-hosted competes.

Example with numbers:

  • 100M tokens/month with gpt-4o-mini ($0.15 + $0.60 / 1M tokens, blended ~$0.35) = $35/month
  • Self-hosted Llama 8B on A10G ($720/month) — doesn't pay off at low volume
  • 1B tokens/month managed = $350/month
  • Self-hosted still $720/month but the investment starts to make sense if you're going to grow

Vector database

Managed (Pinecone, Weaviate Cloud, Qdrant Cloud):

  • Pinecone: $0.096/1M vectors/hour for starter
  • For 1M vectors: ~$70/month
  • For 10M vectors: ~$700/month
  • Setup: 30 min
  • Operations: zero

Self-hosted (ChromaDB, pgvector, Weaviate self-hosted):

  • A server with enough RAM (vectors are memory-hungry)
  • 10M vectors × 1536 dim × 4 bytes = ~60GB RAM
  • A 64GB RAM server on AWS: r5.2xlarge ~$280/month
  • Operations: 4-8 hours/month from the team (backup, scaling, debugging)

Cross-over: managed wins up to ~1-2M vectors. Beyond that, self-hosted with infra costs starts to win if your team has the skills.

Hosting (API)

Managed serverless (Cloud Run, Lambda, Modal):

  • Pay per request + execution time
  • Zero infrastructure overhead
  • Automatic auto-scaling
  • Cold starts (except with min-instances)

Self-hosted containers (ECS, GKE, AKS, plain VMs):

  • Pay per instance (always running)
  • Real DevOps overhead
  • More control
  • No cold starts

For AI services with LLM: serverless works well if your workers are stateless. But LLM workers often have a GPU + model loaded → containers is more common.

Embeddings model

Managed (OpenAI embeddings, Cohere):

  • $0.02/1M tokens (OpenAI ada-002)
  • Auto-scaling, simple

Self-hosted (bge-large, all-MiniLM):

  • An open-source model on your infra
  • Lower latency (no network)
  • No per-request cost

Cross-over: managed wins almost always for embeddings. Self-hosted only if: strict privacy, very high volume (>1B tokens/month), or latency <50ms is critical.

Complete TCO: not just the list price

The most common trap is comparing only the platform's list prices. The real TCO includes:

TCO of Pinecone managed

Pinecone bill ............... $700/month
Engineering time ............ $80/hr × 1hr/month = $80/month
                              (setup already done, occasional maintenance)
Total TCO .................... $780/month

TCO of ChromaDB self-hosted

EC2 r5.2xlarge .............. $280/month
Backup storage (S3) ......... $20/month
Monitoring (Datadog) ........ $50/month
Engineering time ............ $80/hr × 6hr/month = $480/month
                              (scaling, patches, debugging, on-call)
Total TCO .................... $830/month

Conclusion: for 10M vectors, Pinecone wins at these numbers. The crossover would be if Pinecone reached >$1500/month or your team had spare engineering capacity.

The multiplier factor: human time

The human cost is the most underestimated one. Young teams think "1 hour/month of maintenance is nothing". But:

  • That hour accumulates
  • It's an opportunity cost: the engineer isn't building features
  • When there's an incident, it's 8 hours, not 1
  • Onboarding new members requires documentation

Rule: estimate human time generously. 1hr/month of "regular work" + 4hr/month of "occasional incidents" + 8hr/incident when there's an outage.

Hybrid patterns: what's common in production

Few systems are 100% managed or 100% self-hosted. The typical:

Pattern 1: managed for everything, self-hosted for the critical parts

  • Managed: LLM (OpenAI), email (SendGrid), monitoring (Datadog), CI/CD (GitHub Actions)
  • Self-hosted: your API service, your workers, your vector DB (data control)

Pattern 2: dual provider to avoid lock-in

  • Managed primary: Pinecone
  • Self-hosted secondary: ChromaDB with periodic replication
  • If Pinecone fails, fallback to ChromaDB (degradation)

Pattern 3: managed to experiment, self-hosted when scaling

  • Month 1-3: everything managed to validate the product
  • Month 4+: you migrate specific components that justify it (ROI, vendor lock-in, compliance)

When it's clearly managed

  • Small team (<5 engineers), none strong in DevOps
  • Time to production critical (validate the product before optimizing)
  • Low-medium volume (the cross-over isn't reached)
  • Non-strict compliance (you don't need on-prem)

When it's clearly self-hosted

  • Very high and constant volume (>2-3× the managed cost)
  • Strict compliance (HIPAA on-prem, GDPR with specific data residency)
  • Team with DevOps skills available
  • You need features the managed one doesn't offer (custom indexes, fine-tuning, etc.)
  • Vendor lock-in is strategically expensive (e.g., you compete with the vendor)

Common traps

Trap 1 — Only comparing the list price. Pinecone $200/month vs ChromaDB EC2 $150/month looks like self-hosted wins. It's not so if you include ops, monitoring, backups, on-call. Calculate the complete TCO.

Trap 2 — Underestimating engineering time. "My engineer can maintain it in 2 hours/month". Real: 8-15 hours/month including incidents. Estimate 5× what you initially think.

Trap 3 — Self-hosting without SRE skills. Your team doesn't know how to operate Postgres at scale. Migrate to managed RDS. Self-hosted requires skills, not just willingness.

Trap 4 — Migrating from managed to self-hosted before having data. "Pinecone is going to be expensive when we scale." → migrate now. But you're not at scale yet. A premature decision. Wait until you have real numbers.

Trap 5 — Invisible lock-in. Pinecone has specific queries (sparse-dense hybrid, etc.) that ChromaDB doesn't have. If you use them, migration requires refactoring your app. Design with abstractions from the start.

Trap 6 — All-or-nothing mindset. "Everything managed" or "everything self-hosted" is rarely optimal. Component-by-component decisions.

Exercise

Your startup is 6 months old, a team of 4 engineers (one with DevOps), a $50K operational budget/month.

Current stack (all managed):

  • OpenAI gpt-4o-mini: $5K/month
  • Pinecone: $400/month
  • Cloud Run: $300/month
  • Datadog: $200/month
  • Total managed: $5,900/month

Question: should you migrate something to self-hosted? If so, what? In what order?

See solution

Analysis:

  • OpenAI ($5K/month): the most obvious candidate for optimization. Self-hosted?

    • Self-hosted Llama 8B on A100: ~$2K/month infra
      • Engineering time setup (~80hrs) and maintenance (~10hrs/month × $100 = $1000/month)
    • Total: ~$3K/month vs $5K/month managed
    • Savings: ~$2K/month
    • But: quality changes, your product may degrade
    • Recommendation: FIRST TRY aggressive caching (M7-03) and model routing (M7-06). If after optimizations you still spend $4K+/month, consider self-hosted with a good migration plan

  • Pinecone ($400/month): low volume, managed wins. DO NOT migrate.

  • Cloud Run + Datadog: low cost, managed wins. DO NOT migrate.

Optimization order (cost-impact):

  1. First: aggressive caching on OpenAI (expect a 30-40% reduction)
  2. Second: model routing (simple queries to Mistral OR, complex ones to GPT-4o-mini) — expect a 30-50% further reduction
  3. Third: if after that you still spend >$3K/month on LLM, THEN consider self-hosted

Conclusion: DO NOT migrate yet. First optimize managed before paying the migration cost.

Summary

You learned:

  • ✅ A 5-dimension framework for managed vs self-hosted
  • ✅ Complete TCO calculation (including engineering time)
  • ✅ Cross-over points per component: LLM, vector DB, hosting, embeddings
  • ✅ Hybrid patterns (what's common in production)
  • ✅ When it's clearly managed, when it's clearly self-hosted
  • ✅ Traps: list price only, underestimating human time, all-or-nothing

Checkpoint: if for a given component you can calculate the complete TCO and compare it with managed, you're ready.

Next lesson

05 — Serverless vs Containers. Another key trade-off: Lambda/Cloud Run for your workers or ECS with always-running containers? For LLM with GPU, the nuances are specific.

Resources

  1. Pinecone pricing — for your calculations.
  2. AWS EC2 pricing — for self-hosted comparisons.
  3. Total Cost of Ownership frameworks.
  4. Build vs Buy guide — a VC perspective.
  5. Cost of Code (Engineering as a Cost Center) — on human time.
← PreviousCache vs ComputeNext →Serverless vs Containers