5. Cost Modeling for AI Deployment

Overview

In this capsule you'll learn to estimate real deployment costs for AI workloads. Not as a theoretical exercise — with functional calculators, current provider numbers, and models you can adapt to your case. By the end, you'll be able to estimate how much your AI system costs on any deployment strategy before deploying.

Context: "How much will it cost?" is the question that most paralyzes deployment decisions. If you don't know how to estimate, you choose out of fear ("AWS is expensive") or out of ignorance ("Lambda is free"). This capsule gives you tools to decide with data.

Why Cost Modeling Is Different for AI

The hidden cost: API calls to LLMs

In a traditional web app, the main cost is infrastructure: servers, databases, CDN. In an AI app, the main cost is usually the calls to LLM APIs. And that cost multiplies by every user request.

# Cost breakdown per request in a typical AI app
cost_per_request = {
    "infrastructure": 0.00002,  # Lambda or a fraction of a VPS
    "llm_api_call": 0.003,     # GPT-4o-mini, ~500 tokens input + 500 output
    "embedding": 0.0001,       # text-embedding-3-small
    "storage": 0.000001,       # S3 storage per request
}

total_per_request = sum(cost_per_request.values())
# = $0.003121 per request

monthly_requests = 100_000
monthly_cost = total_per_request * monthly_requests
# = $312.10/month

# Of that $312, $300 are LLM API calls
# The infra is $2/month — 0.6% of the total

Insight: For most AI apps, the infrastructure cost (Lambda, VPS, Render) is irrelevant compared to the cost of LLM APIs. Optimizing infra when your OpenAI bill is 50x larger is optimizing the wrong thing.

When infrastructure DOES dominate

The exception is when you use local models (not external APIs): embeddings in memory, open-source models on GPU. There the infra cost (GPU instances, RAM) dominates.

Dominant cost by architecture:

API-based (OpenAI, Anthropic):
  LLM APIs: 90-95% of the total cost
  Infra:     5-10%
  → Optimize PROMPTS, not infra

Self-hosted models (Llama, Mistral):
  GPU compute: 70-85% of the total cost
  Infra:       15-30%
  → Optimize GPU utilization and infra

Hybrid (local embeddings + LLM API):
  LLM APIs: 50-70%
  GPU/RAM:  20-30%
  Infra:    10-20%
  → Optimize both

Cost Model: Fixed vs Variable

Fixed costs

You pay the same regardless of traffic:

fixed_costs_examples = {
    "vps_digitalocean_4gb": 24,      # $/month
    "vps_digitalocean_8gb": 48,      # $/month
    "railway_pro_plan": 5,           # $/month base
    "render_starter": 7,             # $/month
    "domain_name": 1,                # $/month (approx)
    "ssl_certificate": 0,            # Free with Let's Encrypt
    "monitoring_uptimerobot": 0,     # Free tier
}

total_fixed = sum(fixed_costs_examples.values())
# = $85/month (with an 8GB VPS)

Variable costs

They scale with usage:

def variable_costs(monthly_requests: int) -> dict:
    """Calculate variable costs for an AI app."""

    # LLM API (GPT-4o-mini: $0.15/1M input, $0.60/1M output tokens)
    avg_input_tokens = 500
    avg_output_tokens = 300
    llm_cost = monthly_requests * (
        (avg_input_tokens / 1_000_000 * 0.15) +
        (avg_output_tokens / 1_000_000 * 0.60)
    )

    # Lambda (if serverless)
    lambda_cost = monthly_requests * 0.0000169  # 512MB, 2s

    # S3 (storage + requests)
    s3_storage = 0.023  # $/GB/month, assuming 1GB
    s3_requests = monthly_requests * 0.0000004  # GET requests

    # Embeddings (if RAG)
    embedding_cost = monthly_requests * 0.0001  # text-embedding-3-small

    return {
        "llm_api": round(llm_cost, 2),
        "lambda": round(lambda_cost, 2),
        "s3": round(s3_storage + s3_requests, 2),
        "embeddings": round(embedding_cost, 2),
        "total": round(llm_cost + lambda_cost + s3_storage + s3_requests + embedding_cost, 2)
    }

# Example: 100K requests/month
costs = variable_costs(100_000)
# {'llm_api': 25.50, 'lambda': 1.69, 's3': 0.06, 'embeddings': 10.0, 'total': 37.25}

Cost Calculator by Strategy

Scenario parameters

# Define your scenario
scenario = {
    "name": "RAG app for an internal team",
    "monthly_requests": 50_000,
    "avg_duration_seconds": 3,
    "memory_mb": 512,
    "storage_gb": 5,
    "avg_input_tokens": 800,   # RAG context + prompt
    "avg_output_tokens": 400,
    "uses_embeddings": True,
    "team_size": 1,
    "ops_hourly_rate": 50,  # $/hr to calculate ops time
}

Cost by strategy

def calculate_all_strategies(s: dict) -> dict:
    """Calculate the monthly cost for each strategy."""

    # LLM cost (common to all strategies)
    llm_monthly = s["monthly_requests"] * (
        (s["avg_input_tokens"] / 1_000_000 * 0.15) +
        (s["avg_output_tokens"] / 1_000_000 * 0.60)
    )
    embedding_monthly = s["monthly_requests"] * 0.0001 if s["uses_embeddings"] else 0
    api_costs = llm_monthly + embedding_monthly

    strategies = {}

    # LOCAL (VPS)
    vps_price = 24 if s["memory_mb"] <= 4096 else 48
    ops_hours_local = 3  # hrs/month maintenance
    strategies["local"] = {
        "infra": vps_price,
        "ops": ops_hours_local * s["ops_hourly_rate"],
        "api_costs": api_costs,
        "total": vps_price + (ops_hours_local * s["ops_hourly_rate"]) + api_costs
    }

    # SERVERLESS (Lambda)
    gb_seconds = s["monthly_requests"] * s["avg_duration_seconds"] * (s["memory_mb"] / 1024)
    lambda_compute = gb_seconds * 0.0000166667
    lambda_requests = s["monthly_requests"] * 0.0000002
    ops_hours_lambda = 1
    strategies["serverless"] = {
        "infra": round(lambda_compute + lambda_requests, 2),
        "ops": ops_hours_lambda * s["ops_hourly_rate"],
        "api_costs": api_costs,
        "total": round(lambda_compute + lambda_requests + (ops_hours_lambda * s["ops_hourly_rate"]) + api_costs, 2)
    }

    # MANAGED (Railway)
    railway_base = 5
    railway_usage = s["monthly_requests"] * 0.00005  # estimate
    ops_hours_managed = 0.5
    strategies["managed"] = {
        "infra": round(railway_base + railway_usage, 2),
        "ops": ops_hours_managed * s["ops_hourly_rate"],
        "api_costs": api_costs,
        "total": round(railway_base + railway_usage + (ops_hours_managed * s["ops_hourly_rate"]) + api_costs, 2)
    }

    # SELF-HOSTED (EC2)
    ec2_price = 30  # t3.medium on-demand
    ops_hours_selfhosted = 8
    strategies["self_hosted"] = {
        "infra": ec2_price,
        "ops": ops_hours_selfhosted * s["ops_hourly_rate"],
        "api_costs": api_costs,
        "total": round(ec2_price + (ops_hours_selfhosted * s["ops_hourly_rate"]) + api_costs, 2)
    }

    return strategies

results = calculate_all_strategies(scenario)

Expected output for 50K req/month:

LOCAL:       Infra $24    + Ops $150  + APIs $17.00  = $191.00
SERVERLESS:  Infra $1.25  + Ops $50   + APIs $17.00  = $68.25
MANAGED:     Infra $7.50  + Ops $25   + APIs $17.00  = $49.50
SELF-HOSTED: Infra $30    + Ops $400  + APIs $17.00  = $447.00

Winner in total cost: MANAGED ($49.50)
Winner in infra cost: SERVERLESS ($1.25)

Note: The LLM API cost ($17/month) is constant.
The difference is in infra + ops.

The Ops Factor: The Invisible Cost

Operation time by strategy

Operation hours/month (estimate for 1 developer):

LOCAL (VPS + Docker):
├── Monitoring and health checks:     0.5 hrs
├── Updates (OS, Docker, deps):       1.0 hrs
├── Debugging incidents:              1.0 hrs
├── Backups and recovery testing:     0.5 hrs
└── Total:                            3.0 hrs/month

SERVERLESS (Lambda):
├── CloudWatch review:                0.5 hrs
├── Debugging cold starts/timeouts:   0.5 hrs
└── Total:                            1.0 hrs/month

MANAGED (Railway/Render):
├── Dashboard review:                 0.25 hrs
├── Env vars and config updates:      0.25 hrs
└── Total:                            0.5 hrs/month

SELF-HOSTED (EC2 + everything):
├── Patching and security:            2.0 hrs
├── Monitoring and alerts:            1.5 hrs
├── Scaling and capacity planning:    1.0 hrs
├── Networking and firewall:          0.5 hrs
├── Backup and DR:                    1.0 hrs
├── Debugging:                        2.0 hrs
└── Total:                            8.0 hrs/month

If your time is worth $50/hr

LOCAL:       3 hrs × $50 =  $150/month in ops
SERVERLESS:  1 hr  × $50 =   $50/month in ops
MANAGED:     0.5hr × $50 =   $25/month in ops
SELF-HOSTED: 8 hrs × $50 =  $400/month in ops

Insight: For a solo developer, the ops cost in self-hosted ($400/month) far exceeds the infra cost ($30/month). The real cost of self-hosted isn't EC2 — it's your time.

Cost Scenarios: MVP vs Growth vs Scale

MVP (1K requests/day)

Monthly: 30K requests

            Infra    Ops     APIs    TOTAL
LOCAL:      $24      $150    $5      $179
SERVERLESS: $0.50    $50     $5      $55.50
MANAGED:    $0       $25     $5      $30 ← Free tier
SELF-HOST:  $30      $400    $5      $435

Winner: Managed (Railway/Render free tier)

Growth (30K requests/day)

Monthly: 900K requests

            Infra    Ops     APIs    TOTAL
LOCAL:      $24      $150    $153    $327
SERVERLESS: $15      $75     $153    $243
MANAGED:    $50      $25     $153    $228
SELF-HOST:  $30      $400    $153    $583

Winner: Managed (still, but it's getting closer to serverless)

Scale (300K requests/day)

Monthly: 9M requests

            Infra    Ops     APIs     TOTAL
LOCAL:      $48      $200    $1,530   $1,778
SERVERLESS: $150     $100    $1,530   $1,780
MANAGED:    $200     $50     $1,530   $1,780
SELF-HOST:  $60      $600    $1,530   $2,190

Insight at scale: When the API calls dominate ($1,530/month), the infra difference between strategies ($48-$200) is marginal. At scale, optimize your prompts and tokens, not your infra.

Interactive Calculator: Copy and Run

This script is a complete calculator you can copy, adapt to your case, and run locally:

# ai_cost_calculator.py — Copy this file and run it with your data

def ai_deployment_cost_report(
    name: str,
    monthly_requests: int,
    avg_input_tokens: int = 500,
    avg_output_tokens: int = 300,
    llm_model: str = "gpt-4o-mini",
    uses_embeddings: bool = False,
    memory_mb: int = 512,
    avg_duration_s: float = 2.0,
    vps_price: float = 24.0,
    ops_hourly_rate: float = 50.0,
):
    """Generate a cost report for your AI app across all strategies."""

    # LLM model prices ($/1M tokens, update as they change)
    llm_prices = {
        "gpt-4o":      {"input": 2.50, "output": 10.00},
        "gpt-4o-mini": {"input": 0.15, "output": 0.60},
        "claude-sonnet":{"input": 3.00, "output": 15.00},
        "claude-haiku": {"input": 0.25, "output": 1.25},
    }

    prices = llm_prices.get(llm_model, llm_prices["gpt-4o-mini"])
    llm_monthly = monthly_requests * (
        (avg_input_tokens / 1_000_000 * prices["input"]) +
        (avg_output_tokens / 1_000_000 * prices["output"])
    )
    embed_monthly = monthly_requests * 0.0001 if uses_embeddings else 0
    api_total = llm_monthly + embed_monthly

    # Lambda compute
    gb_s = monthly_requests * avg_duration_s * (memory_mb / 1024)
    lambda_cost = gb_s * 0.0000166667 + monthly_requests * 0.0000002

    strategies = {
        "Local (VPS)":    {"infra": vps_price, "ops_hrs": 3},
        "Serverless":     {"infra": round(lambda_cost, 2), "ops_hrs": 1},
        "Managed":        {"infra": round(5 + monthly_requests * 0.00005, 2), "ops_hrs": 0.5},
        "Self-hosted":    {"infra": 30, "ops_hrs": 8},
    }

    print(f"\n{'='*60}")
    print(f"  COST REPORT: {name}")
    print(f"  {monthly_requests:,} requests/month | Model: {llm_model}")
    print(f"{'='*60}")
    print(f"\n  API costs (all strategies): ${api_total:.2f}/month")
    print(f"    LLM: ${llm_monthly:.2f} | Embeddings: ${embed_monthly:.2f}\n")
    print(f"  {'Strategy':<18} {'Infra':>8} {'Ops':>8} {'APIs':>8} {'TOTAL':>10}")
    print(f"  {'-'*18} {'-'*8} {'-'*8} {'-'*8} {'-'*10}")

    for strat, data in strategies.items():
        ops_cost = data["ops_hrs"] * ops_hourly_rate
        total = data["infra"] + ops_cost + api_total
        print(f"  {strat:<18} ${data['infra']:>6.2f} ${ops_cost:>6.0f} ${api_total:>6.2f} ${total:>8.2f}")

    print(f"\n  Note: Ops = hours/month × ${ops_hourly_rate}/hr (your time)")
    print(f"{'='*60}\n")

# === CHANGE THESE VALUES WITH YOUR DATA ===
ai_deployment_cost_report(
    name="My RAG App",
    monthly_requests=50_000,
    avg_input_tokens=800,
    avg_output_tokens=400,
    llm_model="gpt-4o-mini",
    uses_embeddings=True,
    memory_mb=512,
    avg_duration_s=2.0,
    vps_price=24.0,
    ops_hourly_rate=50.0,
)

Run python ai_cost_calculator.py and you get a complete report. Change the parameters to explore scenarios: what happens if you use GPT-4o instead of mini? If you double the traffic?

AI-Specific Cost Optimization

Reducing LLM API cost (90% of the spend)

# Strategy 1: Response caching
import hashlib
import redis

r = redis.Redis()

def cached_llm_call(prompt: str, model: str = "gpt-4o-mini") -> str:
    cache_key = hashlib.md5(f"{model}:{prompt}".encode()).hexdigest()
    cached = r.get(cache_key)
    if cached:
        return cached.decode()  # Cache hit: $0 in API

    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
    )
    result = response.choices[0].message.content
    r.setex(cache_key, 3600, result)  # Cache 1 hour
    return result

# If 30% of requests are repeated, you reduce 30% of the API cost

# Strategy 2: The right model for the task
model_costs = {
    "gpt-4o":      {"input": 2.50, "output": 10.00},  # $/1M tokens
    "gpt-4o-mini": {"input": 0.15, "output": 0.60},   # 16x cheaper
    "gpt-3.5":     {"input": 0.50, "output": 1.50},   # Legacy
}

# If 80% of your requests don't need GPT-4o,
# using gpt-4o-mini for those saves 16x

# Strategy 3: Prompt optimization (fewer tokens)
prompt_v1 = """
You are a helpful AI assistant that specializes in answering
questions about software deployment and cloud infrastructure.
Please analyze the following question carefully and provide
a detailed, comprehensive answer that covers all relevant aspects.

Question: What is Docker?
"""  # ~45 tokens

prompt_v2 = """Answer concisely: What is Docker?"""  # ~8 tokens

# v2 uses 82% fewer input tokens
# At 100K req/month with GPT-4o-mini: savings of $0.56/month (little)
# At 100K req/month with GPT-4o: savings of $9.25/month (significant)

Comparison: When to Use X vs Y

Decision matrix by budget

Troubleshooting

Problem 1: "The OpenAI bill is bigger than the infrastructure one"

Solution: Normal for API-based apps. Prioritize: (1) caching, (2) the right model per task, (3) prompt optimization. The infra is secondary.

Problem 2: "I don't know how to estimate traffic"

Solution: Start with the most flexible strategy (managed or serverless), measure real traffic for 1-2 months, and re-evaluate. Don't optimize for traffic you don't have.

Problem 3: "Costs scale faster than expected"

Solution: Check (1) whether there are loops of LLM calls (a bug can multiply costs), (2) whether the prompt is unnecessarily long, (3) whether you're caching repeated responses. A common bug in LangChain is a retry loop that calls the LLM 5 times per error, multiplying your bill 5x.

Problem 4: "I don't know which LLM model to use to optimize costs"

Solution: Compare prices by task. For most cases:

Task                           | Recommended model      | Cost/100K req
-------------------------------|------------------------|---------------
Simple classification          | gpt-4o-mini            | ~$3
Document summarization         | gpt-4o-mini            | ~$5
Complex reasoning              | gpt-4o                 | ~$600
Code + analysis                | claude-sonnet          | ~$900
Request triage/routing         | claude-haiku           | ~$8

Use the cheapest model that meets the required quality. Evaluate with 50-100 test requests before committing.

Hands-On Exercises

Exercise 1: Calculate your current cost

Use the cost calculator to estimate the monthly cost of your AI app across the 4 strategies. Use real numbers from your app or those from the example scenario.

Exercise 2: Find the breakeven

At how many requests/month does your chosen strategy become more expensive than the alternative? Calculate the inflection point.

Railway: $7.50 fixed + overhead
Lambda: $0.0000169 × requests

Breakeven: $7.50 / $0.0000169 = ~444K requests/month

At <444K req/month: Lambda cheaper in infra
At >444K req/month: Railway may be cheaper (but check plan limits)

Exercise 3: LLM cost optimization

Your app does 100K req/month with GPT-4o (average 800 input + 400 output tokens). Calculate: (a) current cost, (b) cost if you migrate 80% to GPT-4o-mini, (c) cost if you implement caching with a 30% hit rate.

# (a) Current: 100% GPT-4o
current = 100_000 * ((800/1e6 * 2.50) + (400/1e6 * 10.00))
# = 100_000 * (0.002 + 0.004) = 100_000 * 0.006 = $600/month

# (b) 80% GPT-4o-mini, 20% GPT-4o
mini = 80_000 * ((800/1e6 * 0.15) + (400/1e6 * 0.60))  # = $28.80
full = 20_000 * ((800/1e6 * 2.50) + (400/1e6 * 10.00))  # = $120
optimized = mini + full  # = $148.80/month (75% savings)

# (c) Caching 30% hit rate over (b)
requests_after_cache = 100_000 * 0.70  # 70K real requests
mini_cached = 56_000 * ((800/1e6 * 0.15) + (400/1e6 * 0.60))  # = $20.16
full_cached = 14_000 * ((800/1e6 * 2.50) + (400/1e6 * 10.00))  # = $84
with_cache = mini_cached + full_cached  # = $104.16/month (83% total savings)

Exercise 4: Present a budget

Prepare a 3-month budget for your AI app including: infra, APIs, and ops. Include a pessimistic scenario (2x expected traffic).

## Budget: My RAG App (3 months)

### Base scenario (50K req/month)
| Month | Infra (Railway) | APIs (OpenAI) | Ops (my time) | Total |
|-----|-----------------|---------------|-----------------|-------|
| 1   | $7.50           | $17.00        | $25             | $49.50|
| 2   | $7.50           | $17.00        | $25             | $49.50|
| 3   | $7.50           | $17.00        | $25             | $49.50|
| **Total** | **$22.50** | **$51.00**   | **$75**         | **$148.50**|

### Pessimistic scenario (100K req/month)
| Month | Infra | APIs   | Ops  | Total   |
|-----|-------|--------|------|---------|
| 1   | $12   | $34    | $25  | $71     |
| 2   | $12   | $34    | $50  | $96     |
| 3   | $15   | $34    | $50  | $99     |
| **Total** | **$39** | **$102** | **$125** | **$266** |

### Mitigations
- If APIs > $50/month → implement caching
- If infra > $20/month → evaluate a VPS as an alternative
- Monthly review of real costs vs budget

Summary

• The LLM API cost dominates over the infrastructure cost for most AI apps (90%+ of the total).
• Fixed costs (VPS, managed plans) are predictable; variable costs (Lambda, APIs) scale with traffic.
• The ops cost (your time) is the most underestimated cost — self-hosted can cost $400/month in time.
• At scale, the difference between infra strategies is marginal when the APIs dominate. Optimize prompts, not infra.
• The 3 most effective optimizations are: response caching, the right model per task, and prompt optimization.
• Always estimate before deploying. Use this capsule's calculator adapted to your numbers.
• Re-evaluate monthly with real data vs estimates.

Additional Resources

1. AWS Pricing Calculator — Official AWS cost estimator
2. OpenAI Pricing — Up-to-date prices of OpenAI models
3. Anthropic Pricing — Claude prices
4. Cloud Cost Handbook — Vantage — Reference of costs by cloud service
5. Railway Pricing — Detailed Railway pricing
6. Render Pricing — Render pricing