Deploying an LLM model on Modal

Up to now your serverless function called OpenAI via API. That's useful, but it's not what sets Modal apart: for that you have Module 2 (OpenAI directly). What makes Modal special is that you can run the model, not just consume someone else's API.

In this capsule you're going to deploy Mistral 7B Instruct running on an A10G GPU inside a Modal container. You'll see the full flow: image with CUDA, downloading weights, first inference, and persistent cache so the following runs don't download the model from scratch.

By the end you'll be able to:

• Request a specific GPU in the decorator and understand why you chose that one
• Use modal.Volume to persist the model weights between invocations
• Load Mistral 7B with vLLM and generate text
• Measure and understand why the first call takes longer than the following ones

Why it matters

The cold start of a container with basic Python was ~3-8s. With an LLM, the first call can take 60-120s if you download 14GB of weights. If you don't solve this, your endpoint is unviable for real users.

The solution isn't magic: it's understanding when the weights get downloaded and where they're stored. Once this is clear, optimizing is a couple of lines of code.

Choosing a GPU

Modal lets you request a GPU with a string in the decorator:

@app.function(gpu="A10G")
def inference(prompt: str): ...

The options (as of early 2026):

Rule of thumb:

• If your model fits in VRAM with margin → the cheapest GPU that does the job.
• Move to a more expensive GPU only if the cheap one doesn't fit, or if you need more throughput.

Mistral 7B fp16 takes ~14GB. It fits in the A10G (24GB) with margin for activations and KV cache. It doesn't fit comfortably in the T4 (16GB) unless you quantize. That's why we use the A10G.

Check current prices at modal.com/pricing — they change.

The weights problem

If you do this (bad idea):

@app.function(gpu="A10G")
def generate(prompt: str):
    from vllm import LLM
    llm = LLM("mistralai/Mistral-7B-Instruct-v0.3")  # ❌ downloads every time
    return llm.generate(prompt)[0].outputs[0].text

Every call downloads 14GB from Hugging Face. That's:

• Slow (1-3 minutes)
• Expensive (you pay for GPU while it downloads)
• Fragile (Hugging Face could rate-limit you)

The right approach is to separate where the weights live from when they get loaded into GPU:

1. Weights on persistent disk (modal.Volume) → downloaded once, they live between containers.
2. Weights in VRAM → loaded every time the container starts (there's no way to avoid this, but it's fast).
3. Model ready to infer → reused while the container is warm.

Worked example: Mistral 7B with persistent cache

Create mistral.py:

# mistral.py
import modal

app = modal.App("mistral-modal")

# Volume to cache the downloaded weights (persists between runs)
weights_volume = modal.Volume.from_name(
    "mistral-weights", create_if_missing=True
)
CACHE_DIR = "/cache/huggingface"

# Image with vLLM and CUDA dependencies
image = (
    modal.Image.debian_slim(python_version="3.11")
    .pip_install(
        "vllm==0.6.3",
        "huggingface_hub[hf_transfer]==0.26.2",
    )
    .env({"HF_HOME": CACHE_DIR, "HF_HUB_ENABLE_HF_TRANSFER": "1"})
)

MODEL = "mistralai/Mistral-7B-Instruct-v0.3"


@app.function(
    image=image,
    gpu="A10G",
    volumes={CACHE_DIR: weights_volume},
    timeout=600,  # 10 min tolerance for the first download
)
def generate(prompt: str, max_tokens: int = 256) -> str:
    from vllm import LLM, SamplingParams

    llm = LLM(model=MODEL, download_dir=CACHE_DIR)
    sampling = SamplingParams(temperature=0.7, max_tokens=max_tokens)

    # Mistral instruct format: [INST] ... [/INST]
    formatted_prompt = f"[INST] {prompt} [/INST]"
    output = llm.generate(formatted_prompt, sampling)
    return output[0].outputs[0].text.strip()


@app.local_entrypoint()
def main():
    import time

    question = "Explain what FastAPI is in 3 sentences."
    print(f"→ {question}\n")

    start = time.time()
    response = generate.remote(question)
    duration = time.time() - start

    print(response)
    print(f"\n⏱  {duration:.1f}s total (includes cold start if applicable)")

First run:

modal run mistral.py

What to expect:

✓ Building image... (60-90s — installs vLLM with CUDA, heavy)
✓ Starting container (A10G GPU)
✓ Downloading Mistral 7B from Hugging Face (~14GB)... (2-4 min with hf_transfer)
✓ Loading model into GPU (~30s)
✓ Generating...

FastAPI is a modern web framework for Python focused on
building fast, robust APIs. It leverages Python types for
automatic input validation and generates interactive
documentation with no extra effort. Its performance is
comparable to Node.js and Go thanks to being built on top
of Starlette and Pydantic.

⏱  240.3s total

Second run (immediately after):

✓ Container reused (warm)
✓ Generating...

[response]

⏱  3.8s total

Third run (10 minutes later, when the container already shut down):

✓ Starting container (A10G GPU)
✓ Loading model from volume cache (~25s — weights already downloaded)
✓ Generating...

[response]

⏱  31.5s total

Three different regimes:

• Ice-cold cold start (first time): ~4 min — builds image + downloads weights
• Warm (same container alive): ~3-5s — the model is already in VRAM
• Lukewarm cold start (new container, cached weights): ~30s — loads weights from the volume to VRAM

What did each piece do?

Key concept: the Volume isn't the VRAM. It's disk that persists. Modal loads the weights from the volume to the container's disk first, then vLLM sends them to VRAM. The expensive part (downloading from the internet) happens only once in the life of the volume.

Optimizing the lukewarm cold start even further

The ~30s cold start loads weights from the volume to VRAM. There are two next optimizations:

1. @modal.enter() — load the model once per container, not per call.

If your function is invoked 100 times within the same container, you don't want to load Mistral 100 times. Modal lets you initialize resources when the container starts with a lifecycle hook:

@app.cls(
    image=image,
    gpu="A10G",
    volumes={CACHE_DIR: weights_volume},
    timeout=600,
)
class MistralService:
    @modal.enter()
    def load_model(self):
        from vllm import LLM
        self.llm = LLM(model=MODEL, download_dir=CACHE_DIR)

    @modal.method()
    def generate(self, prompt: str, max_tokens: int = 256) -> str:
        from vllm import SamplingParams
        sampling = SamplingParams(temperature=0.7, max_tokens=max_tokens)
        output = self.llm.generate(f"[INST] {prompt} [/INST]", sampling)
        return output[0].outputs[0].text.strip()


@app.local_entrypoint()
def main():
    service = MistralService()
    print(service.generate.remote("What is Modal?"))

@modal.enter() runs only once when the container starts. The following invocations of generate reuse self.llm which is already in VRAM. This is what you really want in production.

2. container_idle_timeout — keep the container warm longer.

By default Modal shuts down idle containers after a few minutes. If you expect burst traffic and want to avoid cold starts, raise it:

@app.cls(
    ...,
    container_idle_timeout=600,  # 10 min of idle before shutting down
)
class MistralService: ...

Trade-off: warm containers charge (a little) for staying on. It's a balance between latency and cost. We go deeper into it in capsule 06.

Common traps

Trap 1 — "Out of memory" when loading the model. Mistral 7B fp16 fits in the A10G (24GB), but barely. If your prompt is very long or max_tokens very high, the KV cache can push you out. Solutions:

• Reduce max_tokens per request.
• Move to L4/A100 (more VRAM).
• Use quantization (AWQ/GPTQ) to reduce the model to ~4GB.

Trap 2 — "The first download took 8 minutes." Without hf_transfer, the download is 3-5× slower. Verify that you have HF_HUB_ENABLE_HF_TRANSFER=1 in image.env(...) and huggingface_hub[hf_transfer] installed.

Trap 3 — "The volume filled up my free quota." Weights are heavy. If you experiment with many models, the volume can grow fast. Modal charges per GB-month of storage. Clean up volumes you don't use:

modal volume list
modal volume delete mistral-weights  # careful: deletes everything

Trap 4 — "Am I forced to use Mistral?" No. Change MODEL to any Hugging Face model compatible with vLLM: meta-llama/Llama-3.1-8B-Instruct, Qwen/Qwen2.5-7B-Instruct, etc. Some models (Llama, Gemma) require accepting the license on HF and passing HF_TOKEN as a secret.

Trap 5 — "vLLM takes a long time importing." vLLM brings large CUDA libs. The first time you build the image it takes 60-90s. After that it's cached. If you rebuild the image often (you change dependencies), consider pinning versions so the cache helps.

Exercise

Modify the code to:

1. Convert it to the version with @app.cls and @modal.enter()
2. Accept a list of prompts in a single call and return the list of responses
3. Measure the time per prompt after the first one (it should drop drastically because the model is already loaded)

# mistral_class.py
import modal
import time

app = modal.App("mistral-class")
weights_volume = modal.Volume.from_name("mistral-weights", create_if_missing=True)
CACHE_DIR = "/cache/huggingface"

image = (
    modal.Image.debian_slim(python_version="3.11")
    .pip_install("vllm==0.6.3", "huggingface_hub[hf_transfer]==0.26.2")
    .env({"HF_HOME": CACHE_DIR, "HF_HUB_ENABLE_HF_TRANSFER": "1"})
)

MODEL = "mistralai/Mistral-7B-Instruct-v0.3"


@app.cls(image=image, gpu="A10G", volumes={CACHE_DIR: weights_volume}, timeout=600)
class MistralService:
    @modal.enter()
    def load(self):
        from vllm import LLM
        self.llm = LLM(model=MODEL, download_dir=CACHE_DIR)

    @modal.method()
    def generate_batch(self, prompts: list[str], max_tokens: int = 256) -> list[str]:
        from vllm import SamplingParams
        sampling = SamplingParams(temperature=0.7, max_tokens=max_tokens)
        formatted_prompts = [f"[INST] {p} [/INST]" for p in prompts]
        outputs = self.llm.generate(formatted_prompts, sampling)
        return [o.outputs[0].text.strip() for o in outputs]


@app.local_entrypoint()
def main():
    prompts = [
        "Explain FastAPI in one sentence",
        "Explain Django in one sentence",
        "Explain Flask in one sentence",
        "Explain Starlette in one sentence",
    ]
    service = MistralService()
    start = time.time()
    responses = service.generate_batch.remote(prompts)
    duration = time.time() - start
    for p, r in zip(prompts, responses):
        print(f"\n→ {p}\n  {r}")
    print(f"\n⏱  {duration:.1f}s for {len(prompts)} prompts ({duration/len(prompts):.1f}s/prompt)")

Summary

You learned:

• ✅ Request a GPU with gpu="A10G" (or T4/L4/A100/H100 as needed)
• ✅ Cache the model weights in modal.Volume (avoids repeated downloading)
• ✅ Configure HF_HUB_ENABLE_HF_TRANSFER=1 for fast downloads
• ✅ Three latency regimes: cold-cold (~minutes), cold-lukewarm (~seconds), warm (~milliseconds)
• ✅ The @app.cls + @modal.enter() pattern to load the model once per container

Checkpoint: if your second inference takes <5s and your dashboard shows the mistral-weights volume with ~14GB, you're ready.

Next capsule

In 05 — REST API with Modal you're going to expose your Mistral as a public HTTP endpoint, with basic authentication and JSON request/response. It's what any external client needs to consume your model. There we stop invoking with modal run and start doing curl https://your-app.modal.run/chat.

Resources

1. Modal — GPU acceleration — all available GPUs and how to choose them.
2. Modal — Volumes — persistent storage.
3. Modal — Class lifecycle — @modal.enter, @modal.exit.
4. vLLM — Quickstart — the inference server we use.
5. Hugging Face — Mistral 7B Instruct v0.3 — the model card.
6. hf_transfer GitHub — accelerated download.