4. Serverless vs Containers for AI

Overview

In this capsule you'll compare in depth the two most common deployment options for AI Engineers: serverless (Lambda) and containers (Docker). Not as abstract categories — with real numbers, comparative code, and scenarios where each option shines or fails. By the end, you'll have the criteria to decide when to use each one in your AI system.

Context: The previous capsules introduced 4 categories and 5 evaluation dimensions. In practice, the most frequent decision AI Engineers face is: "Lambda or Docker?" This capsule goes deep on that comparison with AI-specific context.

The Real Dilemma

Why this comparison dominates

Of the 4 deployment categories, two represent 80% of the real decisions for AI Engineers:

1. Containers (Docker/Docker Compose) — Used for local, staging, and production on a VPS or managed platforms
2. Serverless (Lambda) — Used for event-driven APIs, asynchronous processing, and microservices

Self-hosted with Kubernetes is enterprise-level. Managed platforms (Render, Railway) use containers under the hood. The fundamental decision is: do you package your app as a container that runs all the time, or as a function that runs on demand?

The right question isn't "which is better?"

It's: "which is better for MY workload?" A chat service with streaming needs a different answer than a batch document processor. Let's see why.

Containers for AI: In Depth

How it works

Your AI app is a Docker container that runs continuously. It receives requests, processes them, and responds. The container is always in memory — no cold starts, no artificial timeouts.

# main.py — Containerized AI app
from fastapi import FastAPI
from openai import OpenAI
from pydantic import BaseModel
import os

app = FastAPI()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

class AskRequest(BaseModel):
    prompt: str
    max_tokens: int = 500

@app.get("/health")
def health():
    return {"status": "healthy"}

@app.post("/ask")
def ask(request: AskRequest):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": request.prompt}],
        max_tokens=request.max_tokens
    )
    return {
        "answer": response.choices[0].message.content,
        "tokens": response.usage.total_tokens
    }

# Dockerfile
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

# Build and run
docker build -t ai-api .
docker run -p 8000:8000 -e OPENAI_API_KEY=$OPENAI_API_KEY ai-api

# Test
curl -X POST http://localhost:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"prompt": "What is deployment?"}'

Expected output:

{
  "answer": "Deployment is the process of putting an application...",
  "tokens": 87
}

Advantages for AI

*Limited by the host's hardware

Disadvantages for AI

Serverless (Lambda) for AI: In Depth

How it works

Your code is packaged as a Lambda function. AWS runs it when an event arrives (HTTP request via API Gateway, message in SQS, file uploaded to S3). After running, the environment can be destroyed or reused.

# lambda_handler.py — Serverless AI endpoint
import json
import os

def handler(event, context):
    """Lambda handler for AI inference."""
    from openai import OpenAI  # Import inside the handler for cold start optimization

    client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

    body = json.loads(event.get("body", "{}"))
    prompt = body.get("prompt", "")

    if not prompt:
        return {
            "statusCode": 400,
            "headers": {"Content-Type": "application/json"},
            "body": json.dumps({"error": "prompt is required"})
        }

    remaining_time_ms = context.get_remaining_time_in_millis()
    timeout_seconds = max(1, (remaining_time_ms - 2000) / 1000)

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=500,
        timeout=timeout_seconds
    )

    return {
        "statusCode": 200,
        "headers": {"Content-Type": "application/json"},
        "body": json.dumps({
            "answer": response.choices[0].message.content,
            "tokens": response.usage.total_tokens,
            "remaining_time_ms": remaining_time_ms
        })
    }

Advantages for AI

Disadvantages for AI

Direct Comparison

Detailed table

Cold starts: real measurement

# Cold start measurement for different Lambda configurations
# (These are real measured numbers, not estimates)

cold_start_measurements = {
    "python_hello_world": {
        "memory": "128MB",
        "package_size": "5KB",
        "cold_start": "200-500ms",
        "warm_invoke": "5-20ms"
    },
    "python_openai_sdk": {
        "memory": "256MB",
        "package_size": "5MB",
        "cold_start": "1.5-3s",
        "warm_invoke": "50-100ms"
    },
    "python_langchain_full": {
        "memory": "512MB",
        "package_size": "80MB",
        "cold_start": "5-12s",
        "warm_invoke": "100-200ms"
    },
    "python_ml_model_small": {
        "memory": "1024MB",
        "package_size": "200MB",
        "cold_start": "10-25s",
        "warm_invoke": "200-500ms"
    }
}

Streaming: the deal-breaker for chat

If your AI app needs streaming (showing the response token-by-token like ChatGPT), Lambda has a fundamental problem: it doesn't natively support WebSockets or Server-Sent Events. Workarounds exist (Lambda response streaming via function URLs), but they're limited.

# Containers: native streaming with FastAPI
from fastapi.responses import StreamingResponse

@app.post("/chat/stream")
async def chat_stream(request: AskRequest):
    async def generate():
        stream = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": request.prompt}],
            stream=True
        )
        for chunk in stream:
            if chunk.choices[0].delta.content:
                yield f"data: {chunk.choices[0].delta.content}\n\n"
        yield "data: [DONE]\n\n"

    return StreamingResponse(generate(), media_type="text/event-stream")

# Lambda: streaming is NOT native
# You'd need Lambda function URLs with response streaming (limited)
# or the API Gateway WebSocket API (complex and costly)

Cost Crossover Analysis

At what traffic volume does Lambda stop being cheaper than a container on a VPS? This calculator gives you the answer for your specific case:

def cost_crossover_analysis(
    memory_mb: int = 512,
    duration_seconds: float = 2.0,
    vps_monthly_cost: float = 24.0,
):
    """Calculate the point where Lambda costs the same as a VPS."""

    lambda_price_per_gb_second = 0.0000166667
    lambda_price_per_request = 0.0000002
    memory_gb = memory_mb / 1024

    cost_per_lambda_request = (
        memory_gb * duration_seconds * lambda_price_per_gb_second
    ) + lambda_price_per_request

    crossover_requests = vps_monthly_cost / cost_per_lambda_request
    crossover_daily = crossover_requests / 30

    print(f"=== Cost Crossover: Lambda vs VPS ===")
    print(f"Lambda config: {memory_mb}MB, {duration_seconds}s avg duration")
    print(f"VPS cost: ${vps_monthly_cost}/month")
    print(f"Lambda cost per request: ${cost_per_lambda_request:.7f}")
    print(f"")
    print(f"Crossover: {crossover_requests:,.0f} requests/month")
    print(f"           ({crossover_daily:,.0f} requests/day)")
    print(f"")

    test_volumes = [1_000, 10_000, 50_000, 100_000, 500_000, 1_000_000]
    print(f"{'Requests/month':>15} | {'Lambda':>10} | {'VPS':>10} | {'Winner':>10}")
    print(f"{'-'*15}-+-{'-'*10}-+-{'-'*10}-+-{'-'*10}")
    for vol in test_volumes:
        lambda_cost = vol * cost_per_lambda_request
        winner = "Lambda" if lambda_cost < vps_monthly_cost else "VPS"
        print(f"{vol:>15,} | ${lambda_cost:>8.2f} | ${vps_monthly_cost:>8.2f} | {winner:>10}")

# Example: AI app with 512MB, 2s average duration
cost_crossover_analysis(memory_mb=512, duration_seconds=2.0, vps_monthly_cost=24.0)

Output:

=== Cost Crossover: Lambda vs VPS ===
Lambda config: 512MB, 2.0s avg duration
VPS cost: $24.0/month
Lambda cost per request: $0.0000169
Crossover: 1,420,118 requests/month (47,337 requests/day)

  Requests/month |     Lambda |        VPS |    Winner
---------------+------------+------------+-----------
          1,000 |      $0.02 |     $24.00 |     Lambda
         10,000 |      $0.17 |     $24.00 |     Lambda
         50,000 |      $0.84 |     $24.00 |     Lambda
        100,000 |      $1.69 |     $24.00 |     Lambda
        500,000 |      $8.43 |     $24.00 |     Lambda
      1,000,000 |     $16.87 |     $24.00 |     Lambda

Insight: For most AI apps (which are well below 47K req/day), Lambda is cheaper in pure infrastructure. But remember: the cost of debugging and the operational complexity of Lambda don't show up in this calculator.

Decision Tree: Serverless vs Containers

Does your app need streaming (token-by-token)?
├── YES → Containers (native streaming)
└── NO → Unpredictable traffic (big peaks)?
          ├── YES → Is a cold start >3s acceptable?
          │         ├── YES → Serverless
          │         └── NO → Containers with auto-scaling
          └── NO → Budget < $10/month?
                    ├── YES → Serverless (pay-per-use)
                    └── NO → Do you need state between requests?
                              ├── YES → Containers (+ Redis/DB)
                              └── NO → Either works
                                        (choose by DX preference)

Hybrid Pattern: The Best of Both

In production, many teams use both:

┌──────────────────────────────────┐
│  Container (always running)      │
│  ├── Main FastAPI app            │
│  ├── WebSocket handler           │
│  ├── Streaming endpoint          │
│  └── Redis (state/cache)         │
│                                  │
│  Serverless (on-demand)          │
│  ├── Lambda: process documents   │
│  ├── Lambda: generate embeddings │
│  └── Lambda: send notifications  │
└──────────────────────────────────┘

• Container for the main API: always available, streaming, stateful
• Serverless for asynchronous tasks: batch processing, events, functions that don't need to be always running

When the hybrid pattern makes sense

The hybrid pattern isn't always worth it. It adds complexity (two deployment systems, two CI/CD pipelines, two sets of monitoring). Use it when:

1. You have clearly different workloads: interactive API (container) + batch processing (Lambda)
2. The costs justify it: If the batch processing is sporadic, Lambda saves money vs an idle container
3. Your team can maintain both: If you're 1 developer, the extra complexity may not be worth it

If your whole app is an API that calls an LLM and returns the response, a single container is enough. No over-engineering.

# The main API (container) delegates work to Lambda
import boto3
import json

lambda_client = boto3.client("lambda")

@app.post("/process-document")
async def process_document(file_url: str):
    """API endpoint that delegates processing to Lambda."""
    lambda_client.invoke(
        FunctionName="document-processor",
        InvocationType="Event",  # Asynchronous
        Payload=json.dumps({"file_url": file_url})
    )
    return {"status": "processing", "message": "Document queued for processing"}

AI scenarios where the choice is clear

Troubleshooting

Problem 1: "My Lambda has cold starts of >10 seconds"

Cause: Heavy dependencies (full LangChain, NumPy, pandas) that load on every cold start.

Solution: (1) Reduce dependencies: langchain-openai instead of langchain[all]. (2) Use Lambda Layers to pre-package deps. (3) If the SLA is strict, use Provisioned Concurrency (~$5/month per warm instance).

Problem 2: "My container consumes too much RAM"

Cause: ChromaDB, FAISS, or another vector store loaded entirely in memory.

Solution: (1) Use a VPS with more RAM (DigitalOcean 8GB = $48/month). (2) Evaluate an external vector store (Pinecone, Qdrant) that doesn't consume your container's RAM. (3) Implement lazy loading: load only the indexes you need.

Problem 3: "I don't know if my workload is event-driven or always-on"

Cause: You don't have traffic data yet.

Solution: If you don't know, start with containers (they always work). Measure traffic for 2 weeks. If you see that 80% of the time there are no requests, evaluate serverless. It's easier to migrate from a container to Lambda than the other way around.

Hands-On Exercises

Exercise 1: Classify these workloads

For each AI workload, decide whether it's better serverless or containers:

1. Chatbot with response streaming, 500 users/day
2. PDF processor that extracts data, used 2-3 times/day
3. Embeddings API that keeps a vector store in memory
4. Transcription service that processes audio files uploaded by users
5. AI dashboard that answers queries about business metrics

Exercise 2: Calculate the crossover point

Your AI app has 512MB of memory, 2 seconds of average duration per request. At how many requests/month does Lambda stop being cheaper than a $24/month VPS?

vps_cost = 24  # $/month, fixed

lambda_price_per_gb_second = 0.0000166667
lambda_price_per_request = 0.0000002
memory_gb = 512 / 1024  # 0.5 GB
duration_seconds = 2

cost_per_request = (memory_gb * duration_seconds * lambda_price_per_gb_second) + lambda_price_per_request
# = (0.5 * 2 * 0.0000166667) + 0.0000002
# = 0.0000166667 + 0.0000002
# = ~$0.0000169

crossover = vps_cost / cost_per_request
# = 24 / 0.0000169
# = ~1,420,000 requests/month
# = ~47,000 requests/day

Exercise 3: Design a hybrid system

Your AI system has 3 components: (a) chat with streaming, (b) batch document processing, (c) search API with embeddings. Design what goes in containers and what goes in serverless.

CONTAINERS (always running):
├── (a) Chat API with streaming
│   └── FastAPI + WebSocket, uvicorn
│   └── Reason: native streaming, low latency
│
├── (c) Search API with embeddings
│   └── FastAPI + FAISS/ChromaDB in memory
│   └── Reason: vector store in RAM, fast access
│
└── Redis (shared)
    └── Response cache, session state

SERVERLESS (on-demand):
└── (b) Document processor
    └── Lambda triggered by S3 upload
    └── Reason: sporadic use, event-driven, no state needed

User → API Gateway/Nginx
         ├── /chat/* → Container (streaming)
         ├── /search/* → Container (embeddings)
         └── /upload → S3 → Lambda trigger → processing

Exercise 4: Use the crossover calculator

Modify this capsule's cost_crossover_analysis function for your case: your app uses 1024MB of memory and has an average duration of 4 seconds (RAG with multiple steps). At how many requests/day does Lambda equal a $48/month VPS?

cost_crossover_analysis(memory_mb=1024, duration_seconds=4.0, vps_monthly_cost=48.0)

# Output:
# Lambda config: 1024MB, 4.0s avg duration
# Lambda cost per request: $0.0000669
# Crossover: ~717,000 requests/month (~23,900 requests/day)
#
# With more memory and more duration, the crossover drops.
# At 1024MB and 4s, Lambda passes the VPS at ~24K req/day
# (vs ~47K req/day with 512MB and 2s)

Exercise 5: Cold start mitigation plan

Your AI Lambda has 8-second cold starts (LangChain + embeddings). Your SLA says <3s latency. Propose 3 mitigation strategies.

Summary

• The serverless vs containers decision is the most frequent for AI Engineers.
• Containers win when you need: streaming, state, RAM for models, direct debugging, consistent latency.
• Serverless wins when: variable/low traffic, event-driven processing, zero ops, minimal budget.
• Cold starts are serverless's most important AI-specific factor: from 1s (light SDK) to 25s (model in memory).
• Streaming is a deal-breaker: if you need token-by-token, containers.
• The hybrid pattern combines the best: containers for the main API, serverless for batch/asynchronous tasks.
• The cost crossover depends on your volume: Lambda is cheaper up to ~1.4M req/month for a typical AI workload.

Additional Resources

1. AWS Lambda vs ECS — Choosing the Right Service — Official AWS decision guide
2. Lambda Container Images — Deploy Lambda as a container
3. FastAPI Streaming Responses — Streaming in FastAPI
4. Lambda Power Tuning — Optimize Lambda memory/cost
5. Serverless Land — AWS serverless patterns
6. Docker for AI/ML Workloads — Docker for AI applications