REST API with Modal

In the previous capsule you called Mistral with modal run from your terminal. Useful for experimenting, useless for production: your real chatbot, your web app, your mobile client — none of them speaks "Python with the Modal SDK". They speak HTTP.

In this capsule you're going to turn your Mistral function into a public HTTPS endpoint that anyone can consume with curl, fetch, or any HTTP library. We're going to use FastAPI mounted inside Modal with the @asgi_app decorator, add authentication with an Authorization header, and design the JSON contract.

By the end you'll be able to:

• Expose a Modal function as an HTTPS endpoint with no intermediate servers
• Design a JSON request/response contract with Pydantic
• Protect the endpoint with simple token authentication
• Call your endpoint from curl and from Python like any external API

Why it matters

An LLM behind modal run is an experiment. An LLM behind https://your-app.modal.run/chat is infrastructure. This capsule is the one that takes you from one to the other.

Modal makes this surprisingly simple because it already built the container, the autoscaler, the TLS certificate, and the public URL. You just declare the handler.

Mental model: a function + an HTTP router

Modal gives you two primitives for HTTP:

For something serious use @modal.asgi_app() with a FastAPI() inside — that's what we'll see. The FastAPI layer gives you:

• Routing (/chat, /health, /models)
• Validation with Pydantic
• Automatic documentation (/docs)
• Middleware (auth, CORS, logging)

Modal gives you:

• The container that serves that app
• A public URL with TLS
• Autoscaling

Worked example: Mistral endpoint with auth

Start from the code in capsule 04 (the version with @app.cls) and add the HTTP endpoint to it. Create api.py:

# api.py
import os
import modal
from typing import Annotated
from fastapi import FastAPI, HTTPException, Header
from pydantic import BaseModel, Field

app = modal.App("mistral-api")
weights_volume = modal.Volume.from_name("mistral-weights", create_if_missing=True)
CACHE_DIR = "/cache/huggingface"

image = (
    modal.Image.debian_slim(python_version="3.11")
    .pip_install(
        "vllm==0.6.3",
        "huggingface_hub[hf_transfer]==0.26.2",
        "fastapi[standard]",
    )
    .env({"HF_HOME": CACHE_DIR, "HF_HUB_ENABLE_HF_TRANSFER": "1"})
)

MODEL = "mistralai/Mistral-7B-Instruct-v0.3"


# API contract
class ChatRequest(BaseModel):
    prompt: str = Field(..., min_length=1, max_length=4000)
    max_tokens: int = Field(256, ge=1, le=2048)
    temperature: float = Field(0.7, ge=0.0, le=2.0)


class ChatResponse(BaseModel):
    response: str
    model: str
    generated_tokens: int


# The class that runs with GPU
@app.cls(
    image=image,
    gpu="A10G",
    volumes={CACHE_DIR: weights_volume},
    timeout=600,
    container_idle_timeout=300,
)
class MistralService:
    @modal.enter()
    def load(self):
        from vllm import LLM
        self.llm = LLM(model=MODEL, download_dir=CACHE_DIR)

    @modal.method()
    def generate(self, prompt: str, max_tokens: int, temperature: float) -> dict:
        from vllm import SamplingParams
        sampling = SamplingParams(temperature=temperature, max_tokens=max_tokens)
        formatted_prompt = f"[INST] {prompt} [/INST]"
        output = self.llm.generate(formatted_prompt, sampling)[0]
        return {
            "text": output.outputs[0].text.strip(),
            "tokens": len(output.outputs[0].token_ids),
        }


# Authentication helper
def verify_token(authorization: str | None) -> None:
    expected_token = os.environ.get("API_TOKEN")
    if not expected_token:
        raise HTTPException(500, "API_TOKEN not configured on the server")
    if not authorization or not authorization.startswith("Bearer "):
        raise HTTPException(401, "Missing 'Authorization: Bearer <token>' header")
    if authorization.removeprefix("Bearer ") != expected_token:
        raise HTTPException(403, "Invalid token")


# The ASGI app (FastAPI) that Modal will serve
@app.function(
    image=image,
    secrets=[modal.Secret.from_name("mistral-api-token")],
)
@modal.asgi_app()
def web():
    web_app = FastAPI(title="Mistral API")

    @web_app.get("/health")
    def health():
        return {"status": "ok", "model": MODEL}

    @web_app.post("/chat", response_model=ChatResponse)
    def chat(
        req: ChatRequest,
        authorization: Annotated[str | None, Header()] = None,
    ):
        verify_token(authorization)
        service = MistralService()
        result = service.generate.remote(
            req.prompt, req.max_tokens, req.temperature
        )
        return ChatResponse(
            response=result["text"],
            model=MODEL,
            generated_tokens=result["tokens"],
        )

    return web_app

Setting up the authentication secret

Before deploying, create the secret with your token:

modal secret create mistral-api-token API_TOKEN=some-long-random-token

Generate a decent random token with:

python -c "import secrets; print(secrets.token_urlsafe(32))"

Save it in your local password manager; you'll need it to call the endpoint.

Deploying

So far you used modal run (ephemeral, it shuts down when the script finishes). For a public endpoint you need modal deploy:

modal deploy api.py

Output:

✓ Created objects.
├── 🔨 Created mount /Users/.../api.py
├── 🔨 Created function MistralService.*.
└── 🔨 Created web function web => https://your-user--mistral-api-web.modal.run

✓ App deployed in 8.2s! 🎉

The URL https://your-user--mistral-api-web.modal.run is public, with HTTPS, autoscaling, and always available (though it may have a cold start if nobody used it in the last 5 minutes).

Testing the endpoint

Healthcheck (no auth):

curl https://your-user--mistral-api-web.modal.run/health
# {"status":"ok","model":"mistralai/Mistral-7B-Instruct-v0.3"}

Chat (with auth):

curl -X POST https://your-user--mistral-api-web.modal.run/chat \
  -H "Authorization: Bearer your-token-from-above" \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Explain REST in 2 sentences.", "max_tokens": 200}'

Expected response:

{
  "response": "REST is an architectural style for designing APIs where resources are identified by URLs and manipulated with standard HTTP methods (GET, POST, PUT, DELETE). It's stateless: each request carries all the necessary information, without relying on server-side sessions.",
  "model": "mistralai/Mistral-7B-Instruct-v0.3",
  "generated_tokens": 78
}

Without a token:

curl -X POST https://your-user--mistral-api-web.modal.run/chat \
  -H "Content-Type: application/json" \
  -d '{"prompt": "hi"}'
# {"detail":"Missing 'Authorization: Bearer <token>' header"}

With an invalid token:

curl -X POST https://your-user--mistral-api-web.modal.run/chat \
  -H "Authorization: Bearer xxx" \
  -H "Content-Type: application/json" \
  -d '{"prompt": "hi"}'
# {"detail":"Invalid token"}

From Python (client):

import os
import httpx

BASE_URL = "https://your-user--mistral-api-web.modal.run"
TOKEN = os.environ["MISTRAL_API_TOKEN"]  # put it in your local .env

response = httpx.post(
    f"{BASE_URL}/chat",
    headers={"Authorization": f"Bearer {TOKEN}"},
    json={"prompt": "What is Modal?", "max_tokens": 200},
    timeout=60,
)
print(response.json()["response"])

Automatic documentation

FastAPI inside Modal generates docs automatically:

• Swagger UI: https://your-user--mistral-api-web.modal.run/docs
• ReDoc: https://your-user--mistral-api-web.modal.run/redoc
• OpenAPI JSON: https://your-user--mistral-api-web.modal.run/openapi.json

Open /docs in the browser. You'll see your two endpoints (/health, /chat), the request/response schemas, and you can test them from there.

Trap: authentication works in /docs too. You need to paste your token in the "Authorize" button in the top right (Swagger understands it via the security scheme).

Managing the deployment

modal app list           # see your deployed apps
modal app logs mistral-api   # live logs
modal app stop mistral-api   # shut down the deployment
modal deploy api.py      # re-deploy (overwrites the current version)

Re-deploy is atomic: Modal doesn't shut down the old version until the new one is ready. There's no visible downtime.

Common traps

Trap 1 — "I get a 500 when calling /chat and the log says API_TOKEN not configured." You forgot to associate the secret with the web function. Verify that secrets=[modal.Secret.from_name("mistral-api-token")] is on the correct decorator (on web's, not on MistralService's — the auth is validated in web).

Trap 2 — "The endpoint takes 60s the first time." It's the GPU container cold start (loading Mistral). The healthcheck doesn't trigger it — only /chat which invokes MistralService. Strategies to mitigate in capsule 06.

Trap 3 — "My auth just compares strings — is it secure?" For internal use or a trusted client, yes. For production with several clients:

• Use constant-time comparison to avoid timing attacks: hmac.compare_digest(token, expected)
• Use per-client tokens that are rotatable (not a single shared token)
• Consider OAuth2/JWT if you have several users — but that's another capsule

Trap 4 — "I want /chat accessible from my frontend on another domain (CORS error)." Add CORS middleware:

from fastapi.middleware.cors import CORSMiddleware

web_app.add_middleware(
    CORSMiddleware,
    allow_origins=["https://your-frontend.com"],
    allow_methods=["POST", "GET"],
    allow_headers=["Authorization", "Content-Type"],
)

Trap 5 — "When I call the endpoint, Pydantic rejects prompts of >4000 characters." You limited it yourself in the contract with max_length=4000. It's good practice to avoid abuse and runaway costs. Raise the limit or remove it if your real case needs it.

Trap 6 — "I want token streaming (Server-Sent Events)." Yes, it can be done with Modal + FastAPI using StreamingResponse and a generator. It's a pattern of its own that deserves its own capsule — for now, the endpoint returns the full response at the end.

Exercise

Add a POST /chat-batch endpoint that receives a list of prompts and returns their responses, reusing MistralService.generate_batch (from the capsule 04 solution). Include validation that the list has between 1 and 10 prompts.

# Additional schemas
class BatchRequest(BaseModel):
    prompts: list[str] = Field(..., min_length=1, max_length=10)
    max_tokens: int = Field(256, ge=1, le=2048)
    temperature: float = Field(0.7, ge=0.0, le=2.0)


class BatchResponse(BaseModel):
    responses: list[str]
    model: str


# Add a method to the service
@app.cls(...)
class MistralService:
    @modal.enter()
    def load(self):
        from vllm import LLM
        self.llm = LLM(model=MODEL, download_dir=CACHE_DIR)

    @modal.method()
    def generate_batch(self, prompts: list[str], max_tokens: int, temperature: float) -> list[str]:
        from vllm import SamplingParams
        sampling = SamplingParams(temperature=temperature, max_tokens=max_tokens)
        formatted_prompts = [f"[INST] {p} [/INST]" for p in prompts]
        outputs = self.llm.generate(formatted_prompts, sampling)
        return [o.outputs[0].text.strip() for o in outputs]


# In the web function
@web_app.post("/chat-batch", response_model=BatchResponse)
def chat_batch(
    req: BatchRequest,
    authorization: Annotated[str | None, Header()] = None,
):
    verify_token(authorization)
    service = MistralService()
    responses = service.generate_batch.remote(
        req.prompts, req.max_tokens, req.temperature
    )
    return BatchResponse(responses=responses, model=MODEL)

curl -X POST https://.../chat-batch \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"prompts": ["What is FastAPI?", "What is Modal?"], "max_tokens": 100}'

Summary

You learned:

• ✅ Turn a Modal function into a public HTTPS endpoint with @modal.asgi_app()
• ✅ Design request/response contracts with Pydantic (free validation)
• ✅ Implement auth with a Bearer token using a modal.Secret
• ✅ modal deploy vs modal run — one persists, the other is ephemeral
• ✅ Automatic documentation at /docs (Swagger UI)

Checkpoint: if you can curl from another terminal and get Mistral's response back, you have a working HTTP service.

Next capsule

In 06 — Autoscaling and cold starts we're going to go deeper into what happens when several clients call your endpoint at once. You'll see:

• How Modal spins up multiple containers in parallel
• How to configure concurrency_limit, container_idle_timeout, keep_warm
• Strategies to keep first-byte-latency low in production

It's the capsule that separates "demo deployment" from "deployment ready for real users".

Resources

1. Modal — Web endpoints — all the ways to expose HTTP.
2. Modal — ASGI apps — the pattern with FastAPI.
3. FastAPI — Security — auth beyond the simple Bearer.
4. Pydantic v2 — Field validators — complex validations.
5. Modal — Logs and monitoring — modal app logs.