2. Landscape of Deployment Options

Overview

In this capsule you'll explore in depth the four deployment categories for AI systems: Local, Serverless, Managed, and Self-hosted. Not as abstract concepts — with concrete examples of how each category applies to a real AI app. By the end, you'll be able to explain to a colleague when and why to choose each one.

Context: The previous capsule introduced the 4 categories as an overview. Here you go deep on each one: what it includes, what technologies represent it, how it looks in practice, and — crucially — what kind of AI system fits best in each category. This isn't a catalog of cloud services; it's a thinking framework.

Local Deployment

What it is

Local deployment means your application runs on infrastructure you control directly: your laptop, a server in your office, or a VPS (Virtual Private Server) you rent. You manage the operating system, the dependencies, the networking, and the application's lifecycle.

In the context of this guide, "local" refers mainly to Docker Compose multi-container: your AI app, Redis, a database, all running in containers orchestrated locally. Module 2 goes deep on this.

Representative technologies

Local Deployment Stack:
├── Docker + Docker Compose     → Container orchestration
├── VPS (DigitalOcean, Linode)  → Rented infrastructure
├── Nginx/Caddy                 → Reverse proxy and SSL
├── systemd                     → Process management
└── SSH                         → Remote access

AI-specific example

Imagine a RAG (Retrieval Augmented Generation) app for an internal team of 20 people:

# docker-compose.yml — Local RAG app
services:
  api:
    build: ./api
    ports:
      - "8000:8000"
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - REDIS_URL=redis://cache:6379
    depends_on:
      cache:
        condition: service_healthy

  cache:
    image: redis:7-alpine
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 3

  embeddings:
    build: ./embeddings-service
    volumes:
      - ./data/vectors:/app/vectors

When to choose Local

Quick profile

Cost:           Low-Medium (VPS ~$5-40/month fixed)
Complexity:     Medium (you manage the infra)
Scalability:    Low (manual, vertical)
Control:        High (full access)
Time-to-deploy: Medium (initial setup, then fast)

Common AI use cases in Local

Serverless Deployment

What it is

Serverless means you don't manage servers. You upload your code (as a function or container), and the cloud provider takes care of running it, scaling it, and charging you only for execution time. "Serverless" doesn't mean "no servers" — it means the servers aren't your problem.

The best-known implementation is AWS Lambda, which runs functions in response to events (HTTP requests, queue messages, S3 changes). Module 3 goes deep on Lambda for AI.

Representative technologies

Serverless Stack:
├── AWS Lambda               → Serverless functions
├── API Gateway              → HTTP trigger for Lambda
├── Google Cloud Functions   → Alternative to Lambda
├── Azure Functions          → Microsoft alternative
├── Vercel Functions         → Serverless for frontend
└── AWS Fargate              → Serverless containers (no EC2 to manage)

AI-specific example

An endpoint that receives a prompt and returns an LLM's response:

# lambda_handler.py — Serverless AI endpoint
import json
import os
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

def handler(event, context):
    """Lambda that invokes an LLM and returns the response."""
    body = json.loads(event.get("body", "{}"))
    prompt = body.get("prompt", "")

    if not prompt:
        return {
            "statusCode": 400,
            "body": json.dumps({"error": "prompt is required"})
        }

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=500,
        timeout=25  # Lambda timeout awareness
    )

    return {
        "statusCode": 200,
        "body": json.dumps({
            "answer": response.choices[0].message.content,
            "model": "gpt-4o-mini",
            "tokens": response.usage.total_tokens
        })
    }

When to choose Serverless

Cold starts: the elephant in the room

For AI, cold starts are the most important factor of serverless. When Lambda hasn't run your function recently, it needs to:

1. Provision a container
2. Download your code and dependencies
3. Initialize the Python runtime
4. Import libraries (openai, langchain, numpy)

Typical cold start by function type:
├── "Hello World" in Python:       ~200-500ms
├── API call to OpenAI:            ~1-3s (imports)
├── LangChain + embeddings:        ~5-10s (heavy imports)
└── ML model in memory:            ~10-30s (model loading)

For a chat API, 1-3 seconds of cold start can be acceptable. For a service that needs <200ms latency, it isn't.

Quick profile

Cost:           Variable (pay-per-invocation, can be very low or very high)
Complexity:     Low-Medium (you don't manage infra, but debugging is different)
Scalability:    Very High (automatic, from 0 to thousands)
Control:        Low (you don't control the runtime, memory limits, timeouts)
Time-to-deploy: Fast (deploy in minutes)

Common AI use cases in Serverless

Anti-pattern: Loading ML models in memory inside Lambda. Each cold start reloads the model, the latency is unacceptable, and you pay for the loading time. If you need models in memory, use containers.

Managed Platforms

What it is

Managed platforms (also called PaaS — Platform as a Service) are services that abstract the infrastructure and let you deploy from Git with minimal configuration. You upload your code or Dockerfile, and the platform builds, deploys, and manages SSL, domains, and basic scaling.

The most relevant options in 2026 are Render, Railway, and Fly.io. Module 7 goes deep on these three.

Representative technologies

Managed Platforms:
├── Render          → Deploy from Git, free tier, databases included
├── Railway         → Premium developer experience, add-ons
├── Fly.io          → Edge deployment, multiple regions
├── Heroku          → The OG, more expensive, less recent innovation
├── Google Cloud Run → Managed containers, pay-per-use
└── AWS App Runner  → AWS version of managed containers

AI-specific example

Deploy your FastAPI + AI app on Railway:

# railway.json — Minimal config
{
  "build": {
    "builder": "DOCKERFILE"
  },
  "deploy": {
    "startCommand": "uvicorn main:app --host 0.0.0.0 --port $PORT",
    "healthcheckPath": "/health",
    "restartPolicyType": "ON_FAILURE"
  }
}

# main.py — AI app deployable on any managed platform
from fastapi import FastAPI
from openai import OpenAI
import os

app = FastAPI()
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

@app.get("/health")
def health():
    return {"status": "healthy", "service": "ai-api"}

@app.post("/ask")
def ask(prompt: str):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=500
    )
    return {"answer": response.choices[0].message.content}

# Deploy on Railway (from the terminal)
railway login
railway init
railway up

# In 2-3 minutes: https://your-app.railway.app/health

When to choose Managed

Quick profile

Cost:           Medium (generous free tiers, $5-50/month for production)
Complexity:     Low (deploy from Git, minimal config)
Scalability:    Medium (limited auto-scale, vertical more than horizontal)
Control:        Medium (you configure your app, not the underlying infra)
Time-to-deploy: Very Fast (minutes from git push)

Common AI use cases in Managed

AI consideration: Check the plan's memory limits. ChromaDB with 500K documents needs ~2-4GB of RAM. If your managed plan offers 512MB, it won't fit. Check before choosing.

Self-hosted Deployment

What it is

Self-hosted means you manage the complete infrastructure: servers (physical or cloud VMs), operating system, networking, security, scaling, backups, and everything else. It's maximum control in exchange for maximum operational responsibility.

In practice, self-hosted for AI usually means EC2 instances on AWS (or the equivalent on GCP/Azure) where you install and configure everything. Large companies with DevOps/SRE teams operate this way.

Representative technologies

Self-hosted Stack:
├── EC2/GCE/Azure VMs     → Compute
├── Kubernetes (EKS/GKE)  → Orchestration (advanced)
├── Terraform/Pulumi       → Infrastructure as Code
├── Ansible/Chef           → Configuration management
├── Prometheus/Grafana     → Monitoring
└── Nginx/HAProxy          → Load balancing

AI-specific example

An enterprise AI system that needs GPUs for local inference of its own models:

Self-hosted architecture for AI:
┌─────────────────────────────────┐
│  Load Balancer (Nginx)          │
├─────────────────────────────────┤
│  App Server 1    App Server 2   │  ← EC2 instances
│  (FastAPI)       (FastAPI)      │
├─────────────────────────────────┤
│  GPU Instance (p3.2xlarge)      │  ← For inference
│  (Custom model, embeddings)     │
├─────────────────────────────────┤
│  Redis Cluster   PostgreSQL     │
│  (Cache)         (Data)         │
└─────────────────────────────────┘

When to choose Self-hosted

Quick profile

Cost:           High (VMs, networking, ops team)
Complexity:     Very High (you manage EVERYTHING)
Scalability:    High (horizontal with load balancers, but manual or with K8s)
Control:        Maximum (access to everything: OS, networking, hardware)
Time-to-deploy: Slow (initial setup can take days/weeks)

Comparison: The 4 Categories

Summary table

AI-specific dimensions by category

Quick decision diagram

Do you need GPUs for your own models?
├── YES → Self-hosted (EC2 with GPU)
└── NO → Is your traffic unpredictable (peaks)?
          ├── YES → Serverless (Lambda)
          └── NO → Do you need to be online in <1 hour?
                    ├── YES → Managed (Render/Railway)
                    └── NO → Do you have an ops team?
                              ├── YES → Self-hosted or Local
                              └── NO → Local (dev) + Managed (prod)

Troubleshooting

Problem 1: "I don't know which category applies to my case"

Symptom: You evaluate the 4 categories and they all seem viable or none seems clear.

Solution: Start with the deal-breakers. Answer these 3 questions:

1. Do you need streaming (token-by-token)? → If yes, rule out Serverless
2. Do you need models in memory (>4GB RAM)? → If yes, rule out Serverless and Managed free
3. Is your budget <$30/month? → If yes, rule out Self-hosted

With 1-2 options eliminated, the decision is simpler.

Problem 2: "Serverless seems cheap but my AI app doesn't fit"

Symptom: Lambda is attractive for cost but your app has heavy dependencies, needs state, or the invocations are long.

Solution: Serverless works for AI when the function is light and stateless (API wrapper over an LLM). It doesn't work when you need embeddings in memory, processing >15 min, or streaming. In that case, evaluate Managed or Local.

Lambda works:    prompt → LLM API → response (2-5s, stateless)
Lambda does NOT: query → load embeddings → search → LLM → stream response

Problem 3: "Managed platforms seem limited"

Symptom: Railway/Render seem "too simple" for a serious project.

Solution: Render and Railway serve real production with SLAs. The limit is when you need specific AWS services (SageMaker, Lambda@Edge) or enterprise compliance. For 80% of startups and projects, managed platforms are legitimate production.

Problem 4: "Self-hosted seems more secure by default"

Symptom: You assume that controlling the infrastructure = more security.

Solution: Self-hosted gives you control, but also the responsibility to patch vulnerabilities, configure firewalls, manage SSL certificates, and rotate secrets. A managed platform with a dedicated security team can be more secure than your un-updated VPS. Security isn't control — it's operational discipline.

Problem 5: "My AI app needs a GPU but I don't want to manage infra"

Symptom: You need to run local models (Llama, Mistral) but self-hosted is too much overhead.

Solution: Evaluate managed GPU services: Replicate, Modal, RunPod. They're not the 4 classic categories, but a hybrid. Alternatively, use model APIs (OpenAI, Anthropic) and eliminate the GPU requirement completely — this is what 90% of AI apps do in practice.

Hands-On Exercises

Exercise 1: Classify these scenarios

For each scenario, identify the most appropriate deployment category and justify it:

1. Internal support AI chatbot for a company (50 employees, sensitive data)
2. Image generation API with DALL-E for a startup (growing traffic)
3. RAG system to search technical documentation (internal use, 200 users)
4. Audio transcription service that processes files uploaded by users

Exercise 2: Profile your app

Take an AI app you've built (or the one from the previous bootcamp/guide) and document:

1. What kind of traffic does it have? (constant, variable, peaks)
2. How many users do you expect? (10, 100, 1000, 10K+)
3. Do you need GPUs?
4. Are there data constraints (compliance, on-premises)?
5. What's your monthly budget for infrastructure?

## Profile: My documentation RAG app

- Traffic: Variable, peaks during business hours (9am-6pm)
- Users: ~100 (engineering team)
- GPUs: No (I use the OpenAI API for embeddings and chat)
- Compliance: Internal data, I prefer not to send it to third parties except OpenAI
- Budget: <$30/month

Viable categories:
- Local (Docker Compose on a VPS): $12/month on DigitalOcean, full control
- Managed (Railway): $5-20/month, simpler to operate
- Serverless: Possible but not ideal (I need embeddings in memory)
- Self-hosted: Overkill for 100 users

First impression: Local (Docker Compose on a VPS) for cost/control balance

Exercise 3: Trade-offs in 30 seconds

Without looking at the table, write from memory the main trade-offs of each category in one sentence:

Exercise 4: Quick cost comparison

For each category, estimate the monthly cost of an AI app with these specs: FastAPI + OpenAI API, 5K requests/day, 512MB of RAM needed.

LOCAL (VPS DigitalOcean 2GB):    $12/month fixed
SERVERLESS (Lambda 512MB, 2s):   ~$2.50/month (150K req × $0.0000169)
MANAGED (Railway free tier):     $0/month (fits in the free tier)
SELF-HOSTED (EC2 t3.micro):      $8/month infra + ~$200/month ops time

OpenAI API cost (common to all): 150K req × $0.003 = ~$450/month

Insight: The infra is $0-12/month. The API is $450/month.
The deployment strategy is irrelevant to the total cost.
What matters is optimizing the API calls.

Exercise 5: Your own decision diagram

Create your own decision diagram (different from the one in this capsule) using 3-4 questions that reflect YOUR most important criteria.

Is your budget <$20/month?
├── YES → Do you need a public URL?
│         ├── YES → Managed (Railway free tier)
│         └── NO → Local (Docker Compose)
└── NO → Do you have an ops team (>2 people)?
          ├── YES → Self-hosted (EC2 + K8s)
          └── NO → Unpredictable traffic?
                    ├── YES → Serverless (Lambda)
                    └── NO → Managed (Render Pro)

Summary

• Local deployment runs on your infrastructure (Docker Compose, VPS). High control, low cost, manual scalability.
• Serverless (Lambda) runs code without managing servers. Automatic scaling, pay-per-use, but cold starts and limits.
• Managed platforms (Render, Railway, Fly.io) abstract infrastructure. Fast, simple deploy, but less control.
• Self-hosted gives you full control. Maximum flexibility, but maximum operational complexity.
• For AI workloads, the specific factors are: cold starts (they affect inference), memory (models in RAM), timeouts (prompt chains), and API cost (invocations × duration).
• There's no universal "correct" category. There's a correct category for your case, your constraints, your stage.

Additional Resources

1. AWS Lambda vs EC2 — When to Use Each — Official Lambda documentation with use cases
2. Render Documentation — Render docs for deploying Python apps
3. Railway Documentation — Complete Railway guide
4. Fly.io Documentation — Fly.io docs with a focus on edge deployment
5. Docker Compose Documentation — Official Docker Compose reference
6. Serverless Framework — Framework for deploying serverless functions
7. DigitalOcean Pricing — VPS cost reference
8. Cloud Native Computing Foundation — Ecosystem of cloud native tools