3. Trade-offs by Deployment Strategy

Overview

In this capsule you'll evaluate each deployment strategy across five concrete dimensions: cost, operational complexity, scalability, control, and time-to-deploy. Not as qualitative opinions ("Lambda scales well") but with metrics and numbers you can use to compare. By the end, you'll have a quantitative evaluation table that feeds directly into the project's decision matrix.

Context: The previous capsule introduced the 4 categories. Now you evaluate them rigorously. The difference between "I know what exists" and "I know when to choose each one" is the ability to measure trade-offs. This capsule gives you that ability.

The 5 Evaluation Dimensions

Why 5 dimensions

Decision frameworks that use a single dimension ("which is cheapest?") produce bad decisions. A cheap strategy that doesn't scale, or one that scales but takes weeks to deploy, is useless if your constraints include fast growth or time-to-market.

The 5 dimensions capture the factors that most impact AI deployment decisions:

The "I want it all" trap

You can't maximize all 5 dimensions simultaneously. If you want maximum control (self-hosted), you pay with complexity and time-to-deploy. If you want minimum time-to-deploy (managed), you sacrifice control. Deployment engineering is trade-off management, not the pursuit of perfection.

        Low cost ←──────────→ High control
                     \        /
                      \      /
         TRADE-OFF     \    /     TRADE-OFF
          ZONE          \  /       ZONE
                         \/
              Simplicity vs Flexibility

Dimension 1: Cost

Pricing models

Each strategy has a fundamentally different pricing model:

# Simplified cost model per strategy
def cost_local(monthly_vps_price: float, traffic: int) -> float:
    """Fixed cost. It doesn't matter if you have 100 or 10K requests."""
    return monthly_vps_price  # $12-48/month typical

def cost_serverless(invocations: int, avg_duration_ms: int, memory_mb: int) -> float:
    """Pay-per-use. Scales linearly with traffic."""
    price_per_gb_second = 0.0000166667  # AWS Lambda pricing
    gb_seconds = (invocations * avg_duration_ms / 1000) * (memory_mb / 1024)
    request_cost = invocations * 0.0000002  # $0.20 per 1M requests
    return gb_seconds * price_per_gb_second + request_cost

def cost_managed(plan_price: float, add_ons: float = 0) -> float:
    """Tiered pricing. Fixed price per plan + add-ons."""
    return plan_price + add_ons  # $0 (free) to $25-100/month

def cost_selfhosted(instances: int, instance_price: float, ops_hours: float, ops_rate: float) -> float:
    """Infra cost + cost of the team that manages it."""
    return (instances * instance_price) + (ops_hours * ops_rate)

Real numbers for AI workloads

Scenario: AI app with FastAPI that invokes GPT-4o-mini, 10K requests/day, average duration 2 seconds.

Monthly requests: 300K

LOCAL (DigitalOcean Droplet 4GB):
  VPS:           $24/month
  Total:         $24/month
  Note:          Unlimited traffic, fixed cost

SERVERLESS (AWS Lambda 512MB):
  GB-seconds:    300K × 2s × 0.5GB = 300K GB-s
  Compute:       300K × $0.0000166667 = $5.00
  Requests:      300K × $0.0000002 = $0.06
  Total:         ~$5/month
  Note:          Very cheap at this volume

MANAGED (Railway Pro):
  Plan:          $5/month + $0.000463/vCPU-min
  Estimated:     ~$15-25/month
  Note:          Predictable pricing, includes SSL/domain

SELF-HOSTED (EC2 t3.medium):
  Instance:      $30/month (on-demand)
  Ops time:      4 hrs/month × $50/hr = $200/month
  Total:         ~$230/month
  Note:          The real cost is your team's time

Key insight: At 10K req/day, serverless is the cheapest in infrastructure. But if you include the cost of Lambda debugging time (cold starts, timeout issues), the equation changes.

When cost reverses

Inflection point (Lambda vs VPS):

At 10K req/day:     Lambda $5   vs VPS $24    → Lambda wins
At 100K req/day:    Lambda $50  vs VPS $24    → VPS wins
At 1M req/day:      Lambda $500 vs VPS $48    → VPS wins by a lot

Conclusion: Serverless is cheap for low/variable traffic.
For high, constant traffic, fixed cost wins.

Dimension 2: Operational Complexity

What "operational" includes

Operational complexity isn't just initial setup. It's the ongoing effort of keeping your system running in production:

• Updating dependencies and security patches
• Diagnosing and resolving incidents
• Monitoring health and performance
• Managing secrets and credentials
• Backup and disaster recovery
• Manual scaling (if applicable)

Complexity by strategy

LOCAL:
  Setup:           ████████░░ (8/10) — Docker Compose, networking, SSL
  Maintenance:     ██████░░░░ (6/10) — OS, Docker, dependency updates
  Debugging:       ████░░░░░░ (4/10) — Direct access to logs and containers
  Weekly total:    2-4 hrs/week

SERVERLESS:
  Setup:           ████░░░░░░ (4/10) — Lambda + API Gateway config
  Maintenance:     ██░░░░░░░░ (2/10) — AWS manages the infra
  Debugging:       ████████░░ (8/10) — CloudWatch logs, distributed tracing
  Weekly total:    1-2 hrs/week (if all goes well), 5-10 hrs (if something breaks)

MANAGED:
  Setup:           ██░░░░░░░░ (2/10) — Git push and env vars
  Maintenance:     ██░░░░░░░░ (2/10) — The platform manages almost everything
  Debugging:       ██████░░░░ (6/10) — Logs available but limited
  Weekly total:    0.5-1 hrs/week

SELF-HOSTED:
  Setup:           ██████████ (10/10) — Everything: OS, networking, security, monitoring
  Maintenance:     ████████░░ (8/10) — Patches, scaling, backups, on-call
  Debugging:       ██░░░░░░░░ (2/10) — Full access, direct SSH
  Weekly total:    5-15 hrs/week

The serverless paradox

Serverless has the lowest operational complexity when everything works. But when something breaks (unexpected cold starts, timeout in a prompt chain, memory leak in an invocation), debugging Lambda is significantly harder than debugging a Docker container you can exec into.

Serverless debugging flow:
1. User reports an error
2. You search in CloudWatch logs (non-ideal interface)
3. You identify the invocation that failed
4. You can't reproduce it locally (different environment)
5. You deploy a fix, wait for a cold start, test
6. Total: 30min-2hrs per incident

Local debugging flow:
1. User reports an error
2. docker compose logs api | grep ERROR
3. docker compose exec api python -c "reproduce_bug()"
4. Fix, rebuild, test
5. Total: 5-30min per incident

Dimension 3: Scalability

Types of scaling

Vertical scaling:   More CPU/RAM to the same server
                    Simple but has a physical limit
                    E.g.: DigitalOcean 4GB → 8GB → 16GB

Horizontal scaling: More instances of the same service
                    Complex but no theoretical limit
                    E.g.: 1 container → 3 containers → 10 containers

Auto-scaling:       The system scales automatically based on demand
                    Requires configuration, it's not magic
                    E.g.: Lambda, Cloud Run, Kubernetes HPA

Scalability by strategy

AI-specific: How much scale do you really need?

A common mistake is over-dimensioning scalability. Ask yourself:

# Calculating needed capacity
daily_requests = 10_000
peak_multiplier = 3  # Peak = 3x the average
hours_of_peak = 4    # The 4 hours of highest traffic

peak_requests_per_second = (daily_requests * peak_multiplier) / (hours_of_peak * 3600)
# = 30,000 / 14,400 = ~2 requests/second at peak

# A single FastAPI container with uvicorn handles ~100-500 req/s
# Conclusion: 10K req/day does NOT need auto-scaling

If your AI app serves 10K requests/day, a single container is enough. You don't need Lambda or Kubernetes. Before optimizing scalability, verify that you actually need it.

Dimension 4: Control

What "control" means in deployment

Control is your ability to access, configure, and modify each layer of the stack:

Level of control by layer:

                    Local  Serverless  Managed  Self-hosted
Hardware           ──────  ──────────  ───────  ──────────
  Physical access    No       No         No       Yes*
  GPU selection      No       No         No       Yes

OS / Runtime       ──────  ──────────  ───────  ──────────
  Choose OS          Yes      No         No       Yes
  System packages    Yes      Limited    No       Yes
  Python version     Yes      Partial    Yes      Yes

Networking         ──────  ──────────  ───────  ──────────
  Custom ports       Yes      No         Partial  Yes
  Firewall rules     Yes      Partial    No       Yes
  VPN/VPC            Yes      Yes (AWS)  No       Yes

Application        ──────  ──────────  ───────  ──────────
  Code              Yes      Yes        Yes      Yes
  Config            Yes      Yes        Yes      Yes
  Secrets           Yes      Yes        Yes      Yes

* Self-hosted with bare metal

When you need high control

• Custom models on GPU: You need to choose the GPU type, install CUDA drivers, optimize memory
• Regulatory compliance: Control exactly where the data is, who accesses it
• Performance tuning: Configure kernel params, network buffers, caching layers
• Deep debugging: SSH into the server, strace, tcpdump, profiling

When control is unnecessary overhead

For most AI apps that call external APIs (OpenAI, Anthropic), you don't need control of the OS or the hardware. Your app is a smart proxy: it receives a request, builds a prompt, calls the LLM API, returns a response. For this, managed platforms give enough control.

Dimension 5: Time-to-Deploy

From commit to production

LOCAL (Docker Compose on a VPS):
  git push → SSH → docker compose pull → docker compose up
  Time: 5-15 min (with CI/CD: 3-8 min)
  Initial setup: 2-4 hours (VPS, Docker, domain, SSL)

SERVERLESS (Lambda):
  git push → GitHub Actions → deploy Lambda → update API Gateway
  Time: 2-5 min (with CI/CD)
  Initial setup: 1-3 hours (Lambda, Gateway, IAM, permissions)

MANAGED (Railway/Render):
  git push → auto-build → auto-deploy
  Time: 1-3 min
  Initial setup: 15-30 min (connect repo, env vars, domain)

SELF-HOSTED (EC2 + K8s):
  git push → CI/CD → build image → push to registry → rolling update
  Time: 5-20 min
  Initial setup: 1-5 days (infra, K8s, networking, monitoring)

Time-to-deploy matters more than you think

In AI development, fast iteration is critical. You're tuning prompts, switching models, tuning parameters. If each deploy takes 20 minutes, you do 3 deploys/day. If it takes 2 minutes, you do 20+. Iteration speed directly impacts the quality of your AI system.

Quantitative Evaluation Table

Scoring: 1 (worst) to 5 (best) per dimension

*Serverless cost: 5 at low volume, 2 at high volume

Does Serverless win? Only if all dimensions have the same weight. In your case, maybe control weighs more (because you have sensitive data), or cost weighs less (because your company pays). The project's decision matrix lets you assign weights per dimension.

Concrete example: Weighted scoring step by step

Suppose your app is an AI chatbot with streaming for a team of 100 people. Your priorities:

weights = {
    "Cost":            20,  # Limited budget but not extreme
    "Complexity":      25,  # Only 1 developer, complexity is key
    "Scalability":     10,  # 100 users, you don't need auto-scale
    "Control":         15,  # Internal data, I prefer to control it
    "Time-to-deploy":  30,  # I need to be online this week
}

scores = {
    #                    Local  Serverless  Managed  Self-hosted
    "Cost":            [  4,      5,          4,        1  ],
    "Complexity":      [  3,      4,          5,        1  ],
    "Scalability":     [  2,      5,          3,        4  ],
    "Control":         [  4,      2,          3,        5  ],
    "Time-to-deploy":  [  3,      4,          5,        1  ],
}

options = ["Local", "Serverless", "Managed", "Self-hosted"]
totals = {opt: 0 for opt in options}

for dim, weight in weights.items():
    for i, opt in enumerate(options):
        totals[opt] += weight * scores[dim][i]

for opt in sorted(totals, key=totals.get, reverse=True):
    print(f"{opt:15} → {totals[opt]:>4} / 500")

# Output:
# Managed         →  430 / 500   ← WINNER
# Serverless      →  405 / 500
# Local           →  320 / 500
# Self-hosted     →  175 / 500

But wait — this chatbot needs streaming. Lambda doesn't support native streaming. Even though Serverless has a good numeric score, it has a functional deal-breaker. The scoring gives you the ranking; you apply the judgment.

Final result: Managed (Railway) wins by score AND by functionality (it supports WebSockets). Serverless is ruled out by a deal-breaker, not by score.

Comparison: AI-specific Trade-offs

Cold starts and inference latency

Memory for AI models

Inference latency by strategy

Inference latency — the time from when the user sends a prompt to when they receive the response — depends on multiple factors. But the deployment strategy adds a base latency:

# Total latency breakdown for a typical AI request

latency_breakdown = {
    "local": {
        "network_to_server": "1-50ms",
        "cold_start": "0ms (always running)",
        "app_processing": "10-50ms",
        "llm_api_call": "500-3000ms",
        "total_typical": "600-3100ms",
    },
    "serverless": {
        "network_to_api_gw": "10-50ms",
        "cold_start": "1000-15000ms (first request)",
        "app_processing": "10-50ms",
        "llm_api_call": "500-3000ms",
        "total_cold": "1600-18000ms",   # Unacceptable for chat
        "total_warm": "600-3100ms",      # Comparable to local
    },
    "managed": {
        "network_to_platform": "10-100ms",
        "cold_start": "0ms (paid plan) or 30000ms (free tier sleep)",
        "app_processing": "10-50ms",
        "llm_api_call": "500-3000ms",
        "total_typical": "600-3200ms",
    },
    "self_hosted": {
        "network_to_server": "1-50ms",
        "cold_start": "0ms (always running)",
        "app_processing": "10-50ms",
        "llm_api_call": "500-3000ms (or 50-500ms with a local model)",
        "total_api": "600-3100ms",
        "total_local_model": "100-600ms",  # Advantage with your own model
    },
}

Key insight: For apps that call external LLM APIs (OpenAI, Anthropic), the LLM latency dominates (500-3000ms). The difference between strategies (0-100ms extra) is marginal. The serverless cold start is the exception — those extra 1-15 seconds do get noticed.

For apps with local models (Llama, Mistral on GPU), self-hosted wins in total latency because it eliminates the network call to the LLM.

Troubleshooting

Problem 1: "I don't know which dimension to prioritize"

Solution: Start with the hardest constraint. If your budget is <$20/month, that eliminates self-hosted and restricts managed. If you need <500ms latency, that complicates serverless. The most restrictive constraint reduces options fast.

Problem 2: "All strategies seem viable"

Solution: Good sign — it means your case has no extreme constraints. Choose by time-to-deploy: start with the one that gets you to production fastest (probably managed), and re-evaluate when you hit that strategy's limit.

Problem 3: "My evaluation changed after implementing"

Solution: Normal. The pre-implementation evaluation is an estimate. After implementing, you have real data. Update your evaluation and decision matrix with real data — it's a living tool, not a static document.

Problem 4: "My evaluation's weights feel arbitrary"

Solution: Use the "forced elimination" technique. If you could only optimize ONE dimension, which would it be? That one gets weight 30-40. Then of the remaining ones, which would you sacrifice first? That one gets weight 5-10. Work from the extremes toward the center so the weights reflect real preferences, not made-up numbers.

Problem 5: "I need to explain these trade-offs to someone non-technical"

Solution: Simplify to 3 dimensions: money, effort, speed. "Serverless costs less but is slower at the start. Managed is the fastest but we have less control. Self-hosted gives us full control but is the most expensive in team time." Three sentences, no jargon.

Hands-On Exercises

Exercise 1: Quantitative evaluation of your case

Using the scoring table (1-5), evaluate each strategy for YOUR AI app. Add weights to each dimension according to your priorities.

Weights (sum to 100):
- Cost: 30 (main constraint)
- Complexity: 25 (team of 1 person)
- Scalability: 10 (I don't expect much traffic yet)
- Control: 15 (internal but not regulated data)
- Time-to-deploy: 20 (I need to iterate fast)

Weighted evaluation:
                  Local  Serverless  Managed  Self-hosted
Cost (×30):       120      150        90        30
Complexity(×25):  75       100        125       25
Scalab.(×10):     20       50         30        40
Control(×15):     60       30         45        75
Time-to(×20):     60       80         100       20
TOTAL:            335      410        390       190

Recommendation: Serverless (410) > Managed (390) > Local (335) > Self-hosted (190)

Exercise 2: Find the inflection point

For each pair of strategies, identify the point where one becomes better than the other:

1. Serverless vs Local: at how many requests/month is Lambda more expensive than a VPS?
2. Managed vs Self-hosted: at what scale do managed platforms fall short?

Lambda (512MB, 2s average):
  Cost per request: ~$0.0000167 + $0.0000002 = ~$0.000017
  Break-even vs VPS $24/month: $24 / $0.000017 = ~1.4M requests/month

  At <1.4M req/month: Lambda cheaper
  At >1.4M req/month: VPS cheaper

Railway Pro: ~$20-100/month, but memory limits (8GB max)
If your app needs >8GB RAM (local embeddings, models in memory):
  → Managed falls short
  → Self-hosted with EC2 16GB or 32GB is necessary

Also: if you need >100 constant concurrent connections,
managed platforms may throttle.

Exercise 3: Worst-case scenario

For each strategy, describe the worst-case scenario and how to mitigate it:

Exercise 4: Calculate the total latency

Your RAG app has: (a) network latency of 30ms, (b) app processing of 40ms, (c) embedding search of 100ms, (d) LLM API call of 1500ms average. Calculate the total latency for each strategy, considering cold starts.

base_latency = 30 + 40 + 100 + 1500  # = 1670ms

latency_by_strategy = {
    "Local":       f"{base_latency}ms (no cold start)",
    "Serverless":  f"{base_latency + 5000}ms cold / {base_latency}ms warm",
    "Managed":     f"{base_latency + 50}ms (paid plan, no sleep)",
    "Self-hosted": f"{base_latency}ms (no cold start)",
}

# Local:       1670ms — consistent
# Serverless:  6670ms cold / 1670ms warm — first request problematic
# Managed:     1720ms — marginal routing overhead
# Self-hosted: 1670ms — consistent

# Insight: The LLM API call (1500ms) dominates.
# Only the serverless cold start changes the user experience.

Exercise 5: 1-minute pitch

Prepare a 1-minute pitch explaining to your team why you chose strategy X for your AI app. Include: what you evaluated, what you ruled out, and when to re-evaluate.

Summary

• The 5 dimensions (cost, complexity, scalability, control, time-to-deploy) give you a quantitative framework to evaluate strategies.
• Cost varies dramatically: serverless is cheap at low volume, expensive at high volume. Local has a fixed cost. Self-hosted includes team cost.
• Operational complexity is the most underestimated dimension. Managed platforms minimize it; self-hosted maximizes it.
• Scalability is the most over-valued dimension. Most AI apps don't need auto-scaling — verify your real traffic before optimizing.
• Control matters when you have compliance constraints, GPUs, or deep debugging. For APIs that call LLMs, it's less critical.
• Time-to-deploy impacts your iteration speed. In AI development, iterating fast = a better system.
• The weights change everything. An evaluation without weights is an opinion disguised as a framework.

Additional Resources

1. AWS Pricing Calculator — Estimate costs of AWS services
2. Cloud Cost Handbook — Reference of cloud costs by service
3. Vantage Cost Reports — Cloud cost monitoring tool
4. The Pragmatic Engineer — Platform Teams — Blog on infrastructure decisions
5. InfoQ — Cloud Architecture — Articles on cloud architecture
6. Railway Pricing — Detailed Railway pricing