4. Fly.io: Deployment for AI Apps

Description

In this capsule you'll deploy your AI app on Fly.io. If Render is simplicity and Railway is developer experience, Fly.io is geographic distribution. Fly.io runs your containers as micro-VMs in edge locations around the world: your app can be simultaneously in São Paulo, Frankfurt, Tokyo and Virginia, responding from the location closest to the user. For AI, this means low global latency — something neither Render nor Railway offers natively.

Context: You come from deploying on Render (capsule 02) and Railway (capsule 03). Fly.io is the third and final platform of the module. Its main differentiator — edge deployment — is irrelevant if all your users are in one region, but transformative if you need low global latency. By the end of this capsule you'll have experience with all three platforms and real data for the comparison (capsule 05).

Fly.io: Platform Overview

What Fly.io is

Fly.io turns your Docker containers into micro-VMs (Firecracker) and runs them across its global network of 30+ regions. The pitch is:

• Edge deployment: Your app runs where your users are, not in a centralized region
• Native multi-region: A single configuration deploys to multiple regions
• Micro-VMs (Firecracker): Lighter than containers, boot in ~300ms
• Persistent volumes: SSD storage attached to the VMs
• Anycast networking: A global IP that routes to the nearest point
• Machines API: Programmatic control of each individual VM

Deployment model

Your code + Dockerfile
    ↓ flyctl deploy
Fly.io builds the image
    ↓
Distributes to selected regions
    ├── iad (Virginia)
    ├── gru (São Paulo)
    ├── fra (Frankfurt)
    └── nrt (Tokyo)
    ↓
Anycast IP: your-app.fly.dev
    ↓ Request from Mexico
    → Routes to iad (nearest)

    ↓ Request from Japan
    → Routes to nrt (nearest)

Internal architecture

Internet
    ↓ Anycast
Fly.io Edge (30+ regions)
    ├── Proxy (TLS termination, routing)
    └── Firecracker micro-VM
        └── Your container
            └── Your app (FastAPI + AI)

Each micro-VM has:
- Dedicated CPU (not shared like Render Free)
- Dedicated RAM
- Ephemeral storage (or a persistent volume)
- Private IPv6 for inter-VM communication

Pricing (data updated 2026)

For AI workloads — estimate (1 VM, shared-cpu-1x, 1 GB extra RAM):

• VM: $1.94/month (or free if you use the 3 VMs of the free allowance)
• Extra RAM: ~$6/month
• Volume 1 GB: $0.15/month
• Total: ~$8/month for one region

Multi-region (3 regions, shared-cpu-1x, 1 GB RAM each):

• 3 VMs: $5.82/month
• Extra RAM: ~$18/month
• Total: ~$24/month for global presence

Limitations for AI

When edge deployment matters for AI

Edge deployment matters when:

✅ Users distributed globally
   → An AI chatbot for a company with offices in 5 countries
   → Savings: 100-200ms of latency per request

✅ First-response latency is critical
   → The TLS connection + routing + network hop can add up to 200-400ms
   → Edge reduces this to <50ms

✅ Apps with conversational state (streaming)
   → Each streaming token travels less distance
   → Perceptibly faster UX

❌ All users are in one region
   → You don't need edge if 95% of your traffic is from one country
   → A VM in the nearest region is enough

❌ The LLM latency dominates
   → If your LLM (OpenAI) takes 1-3s, the 200ms of network is irrelevant
   → Edge doesn't improve external API latency

Deploy Step-by-Step: AI App on Fly.io

Step 1: Install and authenticate flyctl

# Install
brew install flyctl
# or
curl -L https://fly.io/install.sh | sh

# Login (opens the browser)
flyctl auth login

# Verify
flyctl auth whoami
# email@example.com

Step 2: Create the app on Fly.io

cd deployment-cloud-guide/module-07/app

# Create app (interactive)
flyctl launch
# ? App name: docusearch-ai
# ? Select region: iad (Virginia) — or the one closest to you
# ? Would you like to set up a PostgreSQL database? No (we'll do it later)
# ? Would you like to set up an Upstash Redis database? No
# ? Would you like to deploy now? No (let's configure first)

This generates a fly.toml:

# fly.toml
app = "docusearch-ai"
primary_region = "iad"

[build]

[http_service]
  internal_port = 8000
  force_https = true
  auto_stop_machines = "stop"
  auto_start_machines = true
  min_machines_running = 0

[[vm]]
  memory = "512mb"
  cpu_kind = "shared"
  cpus = 1

Step 3: Configure fly.toml for AI

# fly.toml — configured for AI workloads
app = "docusearch-ai"
primary_region = "iad"

[build]

[env]
  PLATFORM = "flyio"
  LOG_LEVEL = "info"

[http_service]
  internal_port = 8000
  force_https = true
  auto_stop_machines = "stop"
  auto_start_machines = true
  min_machines_running = 1

  [http_service.concurrency]
    type = "requests"
    hard_limit = 250
    soft_limit = 200

[[http_service.checks]]
  grace_period = "10s"
  interval = "30s"
  method = "GET"
  path = "/health"
  timeout = "5s"

[[vm]]
  memory = "1gb"
  cpu_kind = "shared"
  cpus = 1

Step 4: Configure secrets

# Fly.io uses "secrets" for sensitive variables (encrypted)
flyctl secrets set OPENAI_API_KEY=sk-proj-xxx

# List secrets (names only, no values)
flyctl secrets list
# NAME              DIGEST    CREATED AT
# OPENAI_API_KEY    abc123    2026-03-08

# Non-sensitive variables go in fly.toml [env]

Step 5: Deploy

flyctl deploy

# Output:
# ==> Building image
# --> docker build
# ==> Pushing image
# ==> Creating release
# ==> Monitoring deployment
#  1 desired, 1 placed, 1 healthy, 0 unhealthy
# --> v1 deployed successfully

Step 6: Verify

# URL
flyctl status
# App: docusearch-ai
# Hostname: docusearch-ai.fly.dev
# Machines:
# ID       REGION  STATE   CHECKS
# abc123   iad     started 1 total, 1 passing

# Health check
curl https://docusearch-ai.fly.dev/health
# {"status":"healthy","version":"1.0.0","platform":"flyio"}

# Inference
curl -X POST https://docusearch-ai.fly.dev/ask \
  -H "Content-Type: application/json" \
  -d '{"question": "What is edge computing?", "max_tokens": 200}'

# Logs
flyctl logs
# 2026-03-08T15:45:00Z app[abc123] iad [info] Application startup complete
# 2026-03-08T15:45:05Z app[abc123] iad [info] POST /ask 200 1.5s

# Web dashboard
flyctl dashboard
# Opens https://fly.io/apps/docusearch-ai

Fly.io: Multi-Region Deployment

Add regions

# List available regions
flyctl platform regions
# CODE  NAME                  GATEWAY
# ams   Amsterdam, NL         ✓
# cdg   Paris, France         ✓
# fra   Frankfurt, Germany    ✓
# gru   São Paulo, Brazil     ✓
# iad   Ashburn, Virginia     ✓
# lax   Los Angeles, CA       ✓
# nrt   Tokyo, Japan          ✓
# sin   Singapore             ✓
# syd   Sydney, Australia     ✓
# ... (30+ regions)

# Scale to multiple regions
flyctl scale count 3 --region iad,gru,fra

# Verify
flyctl status
# Machines:
# ID       REGION  STATE
# abc123   iad     started
# def456   gru     started
# ghi789   fra     started

fly.toml for multi-region

# fly.toml
app = "docusearch-ai"
primary_region = "iad"

[http_service]
  internal_port = 8000
  force_https = true
  auto_stop_machines = "stop"
  auto_start_machines = true
  min_machines_running = 1

[[vm]]
  memory = "1gb"
  cpu_kind = "shared"
  cpus = 1

# The fly.toml doesn't define secondary regions
# They're managed with flyctl scale:
flyctl scale count 2 --region iad
flyctl scale count 1 --region gru
flyctl scale count 1 --region fra
# Total: 4 VMs across 3 regions

Intelligent routing

Fly.io automatically routes to the nearest point:

Request from Mexico City → iad (Virginia, ~30ms)
Request from Buenos Aires → gru (São Paulo, ~20ms)
Request from Berlin → fra (Frankfurt, ~10ms)
Request from Tokyo → nrt (if you have a VM there) or iad (fallback)

You don't need to configure anything — anycast routing does it automatically.

Fly.io: Volumes and Persistence

Create a volume

# Create a volume in a region
flyctl volumes create ai_data --size 1 --region iad
# Volume: vol_abc123
# Size: 1 GB
# Region: iad

# List volumes
flyctl volumes list

Mount in fly.toml

# fly.toml
[mounts]
  source = "ai_data"
  destination = "/data"

# In your app, /data is persistent across deploys
import os

DATA_DIR = "/data"

def save_embeddings(embeddings, filename):
    path = os.path.join(DATA_DIR, filename)
    with open(path, "wb") as f:
        import pickle
        pickle.dump(embeddings, f)

def load_embeddings(filename):
    path = os.path.join(DATA_DIR, filename)
    if os.path.exists(path):
        with open(path, "rb") as f:
            import pickle
            return pickle.load(f)
    return None

Important limitation: a volume only exists in one region. If you have VMs in iad, gru, and fra, only the VM in iad can access the iad volume. For data shared across regions, use S3 or Cloudflare R2.

Fly.io: Databases

PostgreSQL on Fly.io

# Create a PostgreSQL cluster managed by Fly.io
flyctl postgres create
# ? App name: docusearch-db
# ? Region: iad
# ? VM size: shared-cpu-1x-256mb
# ? Volume size: 1 GB
# ✅ Created: docusearch-db

# Attach to your app
flyctl postgres attach docusearch-db --app docusearch-ai
# ✅ DATABASE_URL added as a secret

# Connect directly
flyctl postgres connect -a docusearch-db
# postgres=# SELECT version();

Redis on Fly.io (via Upstash)

# Fly.io offers managed Redis via Upstash
flyctl redis create
# ? Name: docusearch-redis
# ? Primary region: iad
# ? Read replicas: (none for now)
# ✅ REDIS_URL set as secret

Fly.io CLI: Essential Commands

# App lifecycle
flyctl launch              # Create a new app (interactive)
flyctl deploy              # Build + deploy
flyctl deploy --local-only # Build locally, push image
flyctl destroy             # Delete app

# Status and monitoring
flyctl status              # Status of the app and machines
flyctl logs                # Logs in real time
flyctl logs --region iad   # Logs from a specific region
flyctl dashboard           # Open the web dashboard

# Secrets
flyctl secrets set KEY=val # Add secret
flyctl secrets list        # List secrets
flyctl secrets unset KEY   # Remove secret

# Scaling
flyctl scale count 3                    # 3 VMs in the primary region
flyctl scale count 2 --region gru       # 2 VMs in São Paulo
flyctl scale vm shared-cpu-1x           # Change the VM type
flyctl scale memory 1024                # Change RAM to 1 GB

# Machines (individual control)
flyctl machine list                     # List all machines
flyctl machine stop abc123              # Stop a machine
flyctl machine start abc123             # Start a machine

# Volumes
flyctl volumes create name --size 1 --region iad
flyctl volumes list

# Databases
flyctl postgres create                  # Create PostgreSQL
flyctl postgres connect -a dbname       # Connect to PostgreSQL
flyctl redis create                     # Create Redis (Upstash)

# Network
flyctl ips list                         # View assigned IPs
flyctl certs list                       # View SSL certificates
flyctl certs add api.your-domain.com    # Add a custom domain

# SSH
flyctl ssh console                      # SSH into a running machine
flyctl ssh console --region gru         # SSH into a machine in a specific region

Troubleshooting

Problem 1: "flyctl deploy fails — health check failing"

Solution: Fly.io verifies that your app responds before finishing the deploy. If the health check fails, the deploy is rolled back.

# Verify that the path is correct in fly.toml
[[http_service.checks]]
  grace_period = "30s"   # Give the startup more time
  interval = "30s"
  method = "GET"
  path = "/health"       # Your real endpoint
  timeout = "10s"

# Debug: see what's happening in the VM
flyctl ssh console
# Inside the VM:
curl http://localhost:8000/health

Problem 2: "App starts but requests fail — secret not found"

Solution: Secrets are injected as environment variables. Verify that they exist:

flyctl secrets list
# If OPENAI_API_KEY is missing:
flyctl secrets set OPENAI_API_KEY=sk-proj-xxx
# The deploy restarts automatically

Problem 3: "Volume not accessible — mount failed"

Solution: Volumes are tied to a region. If your VM is in iad but the volume in gru, it can't mount it.

# Check the volume's region
flyctl volumes list
# ID          REGION  SIZE  CREATED
# vol_abc     iad     1GB   2026-03-08

# The VM must be in the same region
flyctl status
# Verify that at least one machine is in iad

Problem 4: "Multi-region is expensive — I can't pay for 3+ VMs"

Solution: You don't need multi-region. A single VM in the region closest to your users is enough for most cases. Multi-region is for when you have global users AND latency is critical.

# Single region is viable and cheap
flyctl scale count 1 --region iad
# ~$8/month with 1 GB RAM

Problem 5: "Auto-stop kills my app — cold start when it comes back"

Solution: By default, Fly.io stops the VMs after inactivity. For AI apps where cold start matters:

# fly.toml
[http_service]
  auto_stop_machines = "off"     # Never stop
  min_machines_running = 1        # At least 1 VM always active

Hands-On Exercises

Exercise 1: Basic deploy on Fly.io

Deploy your AI app on Fly.io in a single region. Configure secrets and verify the health check.

cd deployment-cloud-guide/module-07/app

# Create app
flyctl launch
# Name: docusearch-ai-fly
# Region: iad
# Deploy now: No

# Edit fly.toml
cat fly.toml
# Verify internal_port = 8000

# Add secrets
flyctl secrets set OPENAI_API_KEY=sk-proj-xxx

# Deploy
flyctl deploy

# Verify
flyctl status
# Machines: 1 running in iad

curl https://docusearch-ai-fly.fly.dev/health
# {"status":"healthy","version":"1.0.0","platform":"flyio"}

curl -X POST https://docusearch-ai-fly.fly.dev/ask \
  -H "Content-Type: application/json" \
  -d '{"question": "What is Fly.io?", "max_tokens": 150}'

# View logs
flyctl logs

Exercise 2: Multi-region deploy and measure latency

Scale your app to 3 regions and measure the latency from different geographic points using response headers.

# Scale to 3 regions
flyctl scale count 1 --region iad   # Virginia
flyctl scale count 1 --region fra   # Frankfurt
flyctl scale count 1 --region nrt   # Tokyo

# Verify
flyctl status
# ID       REGION  STATE
# abc123   iad     started
# def456   fra     started
# ghi789   nrt     started

# Add an endpoint that shows the region
# In main.py:
@app.get("/region")
def get_region():
    return {
        "fly_region": os.environ.get("FLY_REGION", "unknown"),
        "fly_alloc_id": os.environ.get("FLY_ALLOC_ID", "unknown"),
    }

flyctl deploy

# Test: The response includes the region that processed the request
curl https://docusearch-ai-fly.fly.dev/region
# {"fly_region":"iad","fly_alloc_id":"abc123"}

# Fly.io injects FLY_REGION automatically into each VM

# measure_regions.py — Measure latency per region
import requests
import time
import statistics

FLY_URL = "https://docusearch-ai-fly.fly.dev"
REGIONS = ["iad", "fra", "nrt"]


def measure_with_region_header(target_region: str, n: int = 5) -> dict:
    """Fly.io lets you force a region with the fly-prefer-region header."""
    latencies = []
    actual_regions = []

    for _ in range(n):
        start = time.time()
        resp = requests.get(
            f"{FLY_URL}/region",
            headers={"fly-prefer-region": target_region},
            timeout=10,
        )
        elapsed = (time.time() - start) * 1000
        latencies.append(elapsed)

        data = resp.json()
        actual_regions.append(data.get("fly_region", "unknown"))
        time.sleep(0.5)

    return {
        "target_region": target_region,
        "actual_regions": list(set(actual_regions)),
        "avg_latency_ms": round(statistics.mean(latencies), 1),
        "p95_latency_ms": round(sorted(latencies)[int(n * 0.95)], 1),
    }


print("=== Latency per Region ===")
for region in REGIONS:
    result = measure_with_region_header(region)
    print(f"  {region}: avg={result['avg_latency_ms']}ms "
          f"(served from: {result['actual_regions']})")

Exercise 3: Configure a persistent volume for caching

Create a volume on Fly.io and use it to cache inference responses on disk, persisting across deploys.

# Create volume
flyctl volumes create ai_cache --size 1 --region iad

# fly.toml — add mount
[mounts]
  source = "ai_cache"
  destination = "/cache"

# file_cache.py — Cache on a persistent disk
import os
import json
import hashlib
from pathlib import Path

CACHE_DIR = Path("/cache/responses")
CACHE_DIR.mkdir(parents=True, exist_ok=True)


def get_cache_key(question: str) -> str:
    return hashlib.sha256(question.encode()).hexdigest()[:32]


def get_cached(question: str) -> dict | None:
    key = get_cache_key(question)
    path = CACHE_DIR / f"{key}.json"
    if path.exists():
        with open(path) as f:
            return json.load(f)
    return None


def save_to_cache(question: str, response: dict) -> None:
    key = get_cache_key(question)
    path = CACHE_DIR / f"{key}.json"
    with open(path, "w") as f:
        json.dump(response, f, ensure_ascii=False)


def cache_stats() -> dict:
    files = list(CACHE_DIR.glob("*.json"))
    total_size = sum(f.stat().st_size for f in files)
    return {
        "entries": len(files),
        "total_size_kb": round(total_size / 1024, 1),
    }

# In main.py
from file_cache import get_cached, save_to_cache, cache_stats

@app.post("/ask")
async def ask(query: Query):
    cached = get_cached(query.question)
    if cached:
        return Answer(**cached)

    response = await call_openai(query.question, query.max_tokens)
    save_to_cache(query.question, response)
    return Answer(**response)

@app.get("/cache/stats")
def get_cache_stats():
    return cache_stats()

flyctl deploy

# Make requests
curl -X POST $FLY_URL/ask -H "Content-Type: application/json" \
  -d '{"question": "What is Docker?", "max_tokens": 100}'

# Verify cache
curl $FLY_URL/cache/stats
# {"entries": 1, "total_size_kb": 0.8}

# Redeploy — the cache persists
flyctl deploy
curl $FLY_URL/cache/stats
# {"entries": 1, "total_size_kb": 0.8}  ← still there

Exercise 4: Compare cold start — Render vs Railway vs Fly.io

Measure the cold start of each platform (time from when the app was asleep until it responds). Document the results.

# cold_start_comparison.py
import time
import requests


def measure_cold_start(name: str, url: str, wait_minutes: int = 20) -> dict:
    """
    To measure real cold start:
    1. Make sure the app has auto-sleep enabled
    2. Don't send requests during wait_minutes
    3. Then measure the first request
    """
    print(f"\n=== {name} ===")
    print(f"Assuming the app received no requests for {wait_minutes} minutes...")

    start = time.time()
    try:
        resp = requests.get(f"{url}/health", timeout=120)
        elapsed = (time.time() - start) * 1000
        return {
            "platform": name,
            "cold_start_ms": round(elapsed, 1),
            "status": resp.status_code,
            "success": True,
        }
    except requests.exceptions.Timeout:
        return {
            "platform": name,
            "cold_start_ms": 120000,
            "status": "timeout",
            "success": False,
        }


platforms = [
    ("Render (Free)", "https://docusearch-ai.onrender.com"),
    ("Railway (Hobby)", "https://docusearch-ai-production.up.railway.app"),
    ("Fly.io (shared)", "https://docusearch-ai-fly.fly.dev"),
]

print("NOTE: For accurate results, don't send requests to any")
print("platform for 20+ minutes before running this script.\n")

results = []
for name, url in platforms:
    result = measure_cold_start(name, url)
    results.append(result)
    print(f"  Cold start: {result['cold_start_ms']}ms")

print("\n=== Cold Start Summary ===")
print(f"{'Platform':<25} {'Cold Start':>12} {'Status':>10}")
print("-" * 50)
for r in sorted(results, key=lambda x: x["cold_start_ms"]):
    print(f"{r['platform']:<25} {r['cold_start_ms']:>10.0f}ms {str(r['status']):>10}")

## Typical cold start results

| Platform | Cold Start | Notes |
|-----------|-----------|-------|
| **Render Free** | 15,000-45,000ms | Spins up a full container |
| **Render Starter** | N/A (no sleep) | Always running |
| **Railway Hobby** | 2,000-8,000ms | With sleep enabled |
| **Fly.io shared** | 300-2,000ms | Firecracker micro-VM startup |
| **Fly.io (min=1)** | N/A (no sleep) | With min_machines_running=1 |

Fly.io has the fastest cold start because Firecracker micro-VMs
boot in ~300ms vs ~10-30s for full Docker containers.

Summary

• Fly.io is edge deployment: your app runs on micro-VMs distributed globally.
• flyctl is the main tool: flyctl launch + flyctl deploy to deploy, flyctl scale for multi-region.
• Multi-region is the differentiator: one IP, multiple regions, automatic routing to the nearest.
• Firecracker micro-VMs boot in ~300ms — a cold start significantly faster than traditional containers.
• Volumes persist data across deploys, but they're tied to a region — they don't share data across VMs in different regions.
• Usage-based pricing is competitive: ~$8/month for one VM with 1 GB RAM in one region, ~$24/month for 3 regions.
• For AI: Edge matters if you have global users AND network latency is significant vs the LLM latency. If your LLM takes 2s, the 200ms of network is marginal.
• WebSocket supported natively — better than Render for token streaming.

Additional Resources

1. Fly.io Documentation — Complete official documentation
2. Fly.io Regions — Complete list of regions
3. Fly.io Machines API — API for programmatic control of VMs
4. Fly.io Volumes — Persistent storage
5. Fly.io PostgreSQL — Managed PostgreSQL
6. Fly.io Pricing — Detailed pricing and calculator
7. Firecracker — AWS Open Source — The technology behind Fly.io