Autoscaling and cold starts

You have a working endpoint. It works fine when you call it. The serious question is: what happens when 100 users call it at the same time? And when nobody calls it for 4 hours?

That's the conversation of this capsule. You're going to understand exactly how Modal scales (when it spins up containers, how long it keeps them alive, what happens to latency) and how to configure the three levers you have to balance latency vs cost.

By the end you'll be able to:

• Predict how many containers your app will use under different traffic patterns
• Configure max_containers, min_containers, scaledown_window, concurrency
• Design cold start mitigation strategies: warm pool, model snapshot, periodic healthcheck
• Measure P50/P95/P99 latency and diagnose whether they're cold starts or a saturated GPU

Why it matters

This is the capsule where your deployment becomes professional. So far, everything you did works — but only for one user, without measuring anything. In real production:

• A 30s cold start costs you users.
• Keeping 10 containers warm "just in case" burns your credits.
• Without limits, a client with a bug that calls 10,000 times can drain your account.

The configurations in this capsule are the ones that separate "functional demo" from "an endpoint your team deploys with confidence".

How Modal scales (mental model)

Every function or class has a container pool that Modal administers automatically:

No traffic:
[       ]     ← 0 containers, costs nothing

1 request arrives:
[ ★ ]         ← Modal spins up 1 container (cold start)
              ← The request waits ~5-30s and is answered

5 requests arrive in parallel:
[ ★ ★ ★ ]     ← Modal spins up 3-5 containers depending on concurrency
              ← Each one handles 1-N requests

No traffic for X minutes:
[       ]     ← Modal shuts down idle containers, back to 0

The parameters that control that dance:

Note about old names: previous versions of Modal used concurrency_limit, keep_warm, container_idle_timeout, allow_concurrent_inputs. If you see those in old blogs/tutorials, the current names (as of early 2026) are the ones in the table. The semantics are the same.

The four levers, in detail

Lever 1 — max_containers (ceiling)

Without this, a traffic storm can spin up 100 A10G GPUs. Your account drains in minutes.

@app.cls(
    gpu="A10G",
    max_containers=10,
    ...
)
class MistralService: ...

With max_containers=10, if 1000 requests arrive Modal queues the excess. Your P99 latency goes up, but your cost is bounded.

Rule of thumb: always set max_containers on any GPU endpoint. It's not optional in production.

Lever 2 — min_containers (warm pool)

Without this, if nobody calls your endpoint for 5 min, the next call pays the full cold start.

@app.cls(
    gpu="A10G",
    min_containers=1,    # 1 container always warm
    max_containers=10,
    ...
)
class MistralService: ...

With min_containers=1, there's always 1 container alive. The first request finds a ready GPU and a loaded model → response in 1-3s.

Trade-off: you're paying for that container 24/7 even if nobody uses it. For an A10G at ~$1/hr, that's ~$720/month. You only set it if your SLA justifies it.

Lever 3 — scaledown_window (how long it takes to scale down)

By default Modal shuts down idle containers in ~60s. If you expect burst traffic with short valleys:

@app.cls(
    gpu="A10G",
    scaledown_window=600,   # 10 min of grace before shutting down
    ...
)
class MistralService: ...

Useful when traffic arrives in bursts: the container stays warm between bursts, you avoid paying repeated cold starts.

Lever 4 — @modal.concurrent(max_inputs=N)

By default, each container handles 1 request at a time. For a GPU with an LLM, this is optimal (vLLM batches internally, you don't want reentrancy).

For a function without GPU that is I/O-bound (queries external APIs), processing several requests in the same container is much more efficient:

@app.function(
    image=image,
    max_containers=20,
)
@modal.concurrent(max_inputs=100)
def call_openai(prompt: str):
    ...

Here 1 container handles up to 100 concurrent requests (asyncio under the hood). Instead of 100 containers, you use 1.

Rule of thumb:

• GPU LLM: max_inputs=1 (default). Let vLLM do the batching.
• CPU I/O-bound: max_inputs=50-100.
• CPU CPU-bound (heavy processing): max_inputs=1.

Applied to your Mistral API

Let's go back to the MistralService from capsule 05 and configure it for production. Edit api.py:

@app.cls(
    image=image,
    gpu="A10G",
    volumes={CACHE_DIR: weights_volume},
    timeout=600,
    # Autoscaling — the new parameters
    min_containers=1,         # 1 GPU always warm
    max_containers=5,         # cap so you don't burn the account
    scaledown_window=300,     # 5 min of grace before shutting down
)
class MistralService:
    @modal.enter()
    def load(self):
        from vllm import LLM
        self.llm = LLM(model=MODEL, download_dir=CACHE_DIR)

    @modal.method()
    def generate(self, prompt: str, max_tokens: int, temperature: float):
        # ... same as before
        ...

With this config:

Diagnosis: cold start or saturated GPU

When your P99 is high, there are two different causes with different remedies:

How to measure in Modal:

modal app logs mistral-api --tail 200

Look for lines like:

[container-abc123] enter (cold start) — took 28.3s
[container-abc123] handled request in 2.1s
[container-def456] enter (cold start) — took 26.8s

If you see many enter (cold start) during a traffic burst → you're suffering cold starts. If you see few but requests with high queueing → you're short on max_containers.

Advanced pattern: warm pool by schedule

If your product has a very marked business schedule and you don't want to pay for a warm pool at dawn, you can't do it from the decorator (Modal doesn't have built-in scheduling of min_containers). You have two options:

Option A — Periodic healthcheck from an external cron. A job in GitHub Actions or an external cron calls your endpoint every 4 minutes during business hours. That keeps the container "touched" → it doesn't enter scaledown.

Option B — A Modal scheduled function that invokes the class. Modal has @app.function(schedule=modal.Period(minutes=4)). A lightweight function that touches MistralService keeps it warm.

@app.function(schedule=modal.Period(minutes=4))
def keep_warm():
    service = MistralService()
    service.generate.remote("ping", max_tokens=1, temperature=0.0)

That gives you a warm pool almost for free (1 minimal inference every 4 minutes vs a container 24/7).

Strategies to reduce the cold start itself

When a cold start is unavoidable (first time of the day, scaling up from a burst), its duration matters. Three optimizations:

1. Pre-bake the model into the image. By default, vLLM downloads weights to the volume and loads them when the container starts. You can pre-bake the weights inside the image (heavier, but faster startup):

def download_weights():
    from huggingface_hub import snapshot_download
    snapshot_download(MODEL, cache_dir="/cache/huggingface")

image = (
    modal.Image.debian_slim()
    .pip_install("vllm==0.6.3", "huggingface_hub[hf_transfer]==0.26.2")
    .env({"HF_HUB_ENABLE_HF_TRANSFER": "1"})
    .run_function(download_weights)  # ← runs this when building the image
)

Trade-off: the image weighs 14GB more; slower rebuild; but containers start without touching external disk.

2. Memory snapshot (if your model supports it). Modal can take a snapshot of the container's memory after @modal.enter() and restore new containers from that snapshot. Loading the Mistral 7B model goes from ~25s to <5s.

@app.cls(
    ...,
    enable_memory_snapshot=True,
)
class MistralService:
    @modal.enter(snap=True)
    def load(self):
        from vllm import LLM
        self.llm = LLM(model=MODEL, download_dir=CACHE_DIR)

Not all models tolerate a snapshot (some GPU drivers "break" when deserializing). Test with your case.

3. Smaller or quantized models. Mistral 7B AWQ (quantized to 4 bits) weighs ~4GB instead of 14GB. Much faster cold start. Trade-off: you lose some quality. For many cases it's worth it.

Common traps

Trap 1 — "I set min_containers=5 to avoid cold starts and my bill exploded." 5 permanent A10G containers ≈ $3,600/month. If your traffic doesn't justify that permanent capacity, don't pay for it. Consider min_containers=1 + scaling on demand.

Trap 2 — "Without max_containers a buggy client drained my account." It happens. Always set max_containers. Also consider a rate limit per API key (not built-in in Modal; you implement it in the FastAPI handler or in front with an API gateway).

Trap 3 — "My P99 goes from 2s to 20s when there are >5 users." Likely: you reached max_containers, the requests are in queue. Raise the cap or add an explicit rate limit.

Trap 4 — "Memory snapshot gives me weird errors." Some GPU drivers don't tolerate snapshots. If you see errors like "CUDA out of memory" or "context invalid" when restoring, disable enable_memory_snapshot.

Trap 5 — "min_containers doesn't seem to work — I still see cold starts." Verify that you deployed after changing the parameter. modal run doesn't apply min_containers. You need modal deploy api.py and to wait for the confirmation.

Trap 6 — "My Modal healthcheck returns 200 but /chat fails with a long cold start." The /health healthcheck doesn't touch the GPU class. For the warm pool to apply to MistralService you need something to call the class's method, not an unrelated endpoint. Use the keep_warm pattern with @app.function(schedule=...) above.

Exercise

Your product is going to have a simulated traffic pattern like this:

• From 9am to 6pm business hours: 100 req/min with bursts up to 300 req/min
• From 6pm to 9am: <5 req/min, occasional spikes

Configure MistralService to:

1. Have low P50 latency during business hours (ideally <3s)
2. Not pay for a GPU running all night
3. Bound the maximum cost (don't scale to 50 GPUs for a burst)

Justify each parameter chosen (don't just put numbers: say why).

@app.cls(
    image=image,
    gpu="A10G",
    volumes={CACHE_DIR: weights_volume},
    timeout=600,
    min_containers=0,          # 0 off-hours, we don't pay for a GPU at night
    max_containers=8,          # cap: covers moderate bursts without risk of runaway spend
    scaledown_window=900,      # 15 min of grace — nearby bursts don't pay cold start
)
class MistralService: ...


# Warm pool by schedule, using schedule
import datetime

@app.function(
    schedule=modal.Period(minutes=4),
    # Modal doesn't allow filtering by hour yet; the filter goes in code.
)
def keep_warm():
    from datetime import datetime
    hour = datetime.utcnow().hour  # adjust to your timezone
    # Only keep warm from 14:00 to 22:00 UTC (≈ 9am-5pm Mexico City time)
    if 14 <= hour < 22:
        service = MistralService()
        service.generate.remote("ping", max_tokens=1, temperature=0.0)

Summary

You learned:

• ✅ How Modal scales: a container pool that grows/shrinks with traffic
• ✅ Four levers: max_containers, min_containers, scaledown_window, @modal.concurrent
• ✅ Diagnosing cold start vs saturated GPU from logs
• ✅ Warm pool patterns: permanent, by schedule, snapshot
• ✅ Cold start optimizations: pre-bake image, memory snapshot, quantized models

Checkpoint: if you can look at your deployment and argue why you chose each parameter (not just say "I left it on default"), you're ready.

Next capsule

07 — Cost optimization covers the other side of the coin: how much exactly does your deployment cost? You'll learn to calculate cost-per-request, choose a GPU based on throughput, and apply batching to reduce the bill without sacrificing latency.

It's the capsule you need before showing numbers to a cofounder or PM who asks "how much does it cost us to serve 1M requests a month?"

Resources

1. Modal — Scaling — all the official scale parameters.
2. Modal — Cold start optimization — official techniques to reduce cold start.
3. Modal — Memory snapshots — fast container restoration.
4. Modal — Schedules — cron jobs and warm pool by schedule.
5. vLLM — Continuous batching explained — why max_inputs=1 is fine with LLMs.