Module 6: Modal — Serverless deployment of LLMs
Module 6: Modal — Serverless deployment of LLMs
You've reached the module where you'll learn to deploy a language model to production without having to learn Kubernetes, without spinning up a GPU cluster, and without paying for a machine running 24/7.
So far, you know three ways to access an LLM:
- OpenAI API (Module 2): managed in the cloud, you pay per token, someone else operates the infrastructure.
- LM Studio / Ollama (Modules 3-4): self-hosted, runs on your machine or on a server you administer.
- OpenRouter (Module 5): an aggregator that gives you access to multiple managed providers with a single API.
You're missing a fourth way that's growing fast in AI startups: serverless with GPU. And that's Modal.
By the end of this module you'll be able to:
- Decide when Modal beats managed cloud and self-hosted (it's not always, and that matters)
- Deploy a Python function with GPU access in less than 5 minutes
- Serve an open-source model (Mistral 7B) as a REST endpoint with autoscaling
- Calculate the real cost of your deployment for different traffic patterns
- Optimize cold starts and cost for production
Where are we in the path?
You're in Module 6 of 8. So far you've covered:
| Module | What it taught you |
|---|---|
| 1 | Framework for deciding which provider to use |
| 2 | OpenAI API (cloud managed) |
| 3 | LM Studio (local with GUI) |
| 4 | Ollama (local with CLI + Docker) |
| 5 | OpenRouter (multi-model aggregator) |
| 6 | Modal (serverless with GPU) ← you are here |
| 7 | Quantitative comparison of all the options |
| 8 | Unified client that abstracts providers |
After this module you have one more option in your toolbox and you'll be ready for the quantitative comparison in Module 7, where you benchmark all the options with real data.
The mental map: the fourth quadrant
To understand Modal, place it on this map:
| You don't operate infrastructure | You operate infrastructure | |
|---|---|---|
| Closed models (GPT-4, Claude) | OpenAI / Anthropic / OpenRouter | (not applicable — they're closed) |
| Open-source models (Mistral, Llama) | Modal, Replicate, RunPod | Ollama, LM Studio, vLLM on your server |
The quadrant "I don't operate infrastructure + I run open-source models that I pick" is where Modal lives. Before these platforms existed, if you wanted to run Llama 3 in production you had to spin up your own GPU on AWS, install CUDA drivers, configure inference servers, handle autoscaling... a lot of DevOps work.
Modal makes a specific promise: "write Python with a decorator, we take care of the rest". Your inference code doesn't change. What changes is that your function lives on infrastructure that doesn't exist until someone invokes it — and when it finishes, it stops existing and stops charging you.
This is called serverless with GPU. "Serverless" doesn't mean there are no servers (there are, in data centers), it means you don't administer them and you don't pay when you're not using them.
A scenario that illustrates the module
Imagine you're an AI Engineer at a 6-person startup. The product is a technical support chatbot that uses a fine-tuned open-source model (you don't want to send customer conversations to OpenAI because of a contractual restriction with enterprise clients).
Traffic has this pattern:
- 9am-6pm business hours: 200-400 requests/minute
- 6pm-9pm: 50-80 requests/minute
- 9pm-9am: practically 0
- Weekends: 10% of a business day's traffic
You have three paths:
Path 1 — Self-hosted (Ollama on a VM with GPU): You rent an A100 on AWS. It costs ~$2.50/hour running 24/7 = ~$1,800/month. It works, but you pay for the full GPU on Sunday at dawn when nobody's using it. And if one day traffic spikes to 1,000 req/min, your single GPU saturates.
Path 2 — Managed (OpenAI): Ruled out. The contractual restriction blocks you.
Path 3 — Serverless with GPU (Modal): You define your inference function, Modal runs instances only when requests arrive. During business hours it scales to 4 instances in parallel if needed. At dawn, zero instances running. You pay for seconds of GPU used, not for reserved hours.
The rough calculation (with Modal's public numbers as of early 2026): if your average request takes 800ms of GPU and you process ~1.5M requests/month, you end up paying between $300-600/month depending on the GPU chosen. Three to six times less than the self-hosted path, with autoscaling for free.
But — and this matters — if traffic were constant at 200 req/min around the clock, self-hosted would win. Modal charges for usage; with saturated usage, "per usage" ends up being more expensive than "flat rent for the GPU".
This is exactly the kind of decision that Module 1 (Decision Framework) taught you to make. Modal isn't better in the abstract — it's better for variable or unpredictable traffic.
Why does this module matter professionally?
Three concrete reasons:
1. AI startups use it in production. Companies like Suno, Lexica, Sieve, and many startups from the latest Y Combinator batch run their inference on Modal or similar competitors (Replicate, RunPod, Beam). If you work at an AI startup in 2026, there's a good chance you'll have to deploy something serverless.
2. It's the only realistic path for deploying OSS models without a DevOps team. If you work alone or on a small team, setting up Kubernetes with GPU operators, CUDA drivers, autoscaling and observability isn't viable. Modal lets you skip all of that. This is a real differentiator in interviews — knowing how to deploy OSS inference without fighting infrastructure.
3. It's the pattern that will most resemble the future. AWS Lambda popularized serverless in 2014. GPU-serverless is in its phase equivalent to Lambda in 2016: emerging but clearly the direction. Learning Modal gives you the mental model for any similar platform that shows up.
Module map
| Capsule | Topic | What you'll build / learn |
|---|---|---|
| 01 | Introduction (this capsule) | Mental model + deciding when to use Modal |
| 02 | Account and CLI setup | Free account, configured CLI, first modal token |
| 03 | Your first serverless function | Remote Python function with a decorator, you observe cold start |
| 04 | Deploying an LLM model | Mistral 7B with vLLM, A10G GPU, loading weights into the image |
| 05 | REST API with Modal | Public HTTP endpoint with basic authentication |
| 06 | Autoscaling and cold starts | Container reuse, keep-warm, GPU pinning to reduce latency |
| 07 | Cost optimization | Real per-request cost calculation, GPU sizing, batching |
| 08 | Project: scalable API | Production-ready endpoint with basic monitoring |
The flow is: you understand the platform (01) → you connect (02) → you run a hello-world (03) → you scale to a real LLM (04) → you expose it as an API (05) → you tune it for production (06-07) → you close with an integrable project (08).
Connection with the path's final project
Module 8 of the path builds a Unified AI Client that abstracts providers. Your Modal deployment from this module will be one of the providers behind that abstraction:
client = UnifiedAIClient(provider="modal", fallback=["openrouter", "ollama"])
response = client.chat("What is FastAPI?")By the time you reach Module 8, you'll already have your Modal endpoint working — you'll just write the adapter. That's why this module ends with a robust endpoint, not a demo.
What this module does NOT cover
To keep the scope clear:
- ❌ Fine-tuning models on Modal. Modal supports training; this module covers inference only. Fine-tuning lives in the Advanced LLM Techniques guide.
- ❌ Advanced cluster autoscaling (multi-region, complex GPU pooling). We see basic autoscaling, not enterprise orchestration.
- ❌ Quantitative comparison against the other providers. That's Module 7, where you benchmark everything with real data.
- ❌ Building your own inference server with vLLM/TGI from scratch. We use vLLM as a partially closed box; going deep into kernel-level optimization is out of scope.
- ❌ Equivalent alternatives (Replicate, RunPod, Beam). We mention when to choose them, we don't cover them hands-on. Modal's mental model transfers to them.
Common traps while taking this module
Four misunderstandings worth avoiding:
Trap 1 — "Serverless is free." It isn't. "Serverless" only means you don't pay for idle infrastructure. When your function runs, you pay seconds of GPU and that adds up fast if your model is slow or your traffic is high. Before deploying to production, measure cost-per-request (we see it in capsule 07).
Trap 2 — "Modal competes with OpenAI." No. OpenAI sells you access to a closed model (GPT-4, etc.). Modal sells you infrastructure to run the model you choose (Mistral, Llama, your fine-tune). They're different products. If you want GPT-4, you don't use Modal. If you want to run an OSS model without spinning up your own infra, yes.
Trap 3 — "More GPU = better." The biggest GPU isn't always the answer. An A10G ($) can serve Mistral 7B perfectly well. Moving to an H100 ($$$$) only makes sense if your model doesn't fit on the A10G or you need very high throughput. Capsule 04 teaches you to choose a GPU based on the model.
Trap 4 — "Cold start doesn't matter."
It does matter, a lot. If your chatbot takes 30 seconds to respond the first time because it has to download 14GB of weights, you lost the user. Capsule 06 covers strategies: container reuse, keep_warm, pre-warmed pools.
Self-assessment question
Before moving on to capsule 02, make sure you can answer:
- What's the difference between managed (OpenAI), self-hosted (Ollama on your server) and serverless with GPU (Modal)? In one sentence each?
- For a workload with constant 24/7 traffic, which option is probably cheaper: Modal or a self-hosted server with Ollama? Why?
- If a startup needs to run Llama 3 70B and has no DevOps team, which path makes more sense and why?
- What does cold start mean in the context of serverless? What causes it when you serve an LLM?
If you're unsure about any of them, it's worth rereading the "A scenario that illustrates the module" section above before continuing. If you feel comfortable answering everything, you're ready.
Answer guide
- Managed: someone runs and operates the model, you call an API and pay per usage. Self-hosted: you spin up and administer the machine that runs the model. Serverless with GPU: someone provides the on-demand infrastructure, you write the code and pay for execution time, without operating anything.
- Self-hosted usually wins with constant 24/7 traffic because the GPU is saturated, and "paying for the full GPU" ends up being more efficient than "paying per usage" when usage is maxed out. Modal wins with variable traffic.
- Modal (or serverless equivalents). Llama 3 70B requires large GPUs; without a DevOps team, spinning up and administering large GPUs on AWS/GCP is very costly in human time.
- Cold start = the time from when a request arrives until your function starts responding, when there was no instance running. In LLMs it's dominated by: starting the container, downloading the model weights, loading them onto the GPU. It can range from seconds to minutes depending on the model.
Evidence of success at the end of the module
You'll know you succeeded if:
- You have a working Modal account and an authenticated CLI
- You deployed a public REST endpoint that serves Mistral 7B with GPU
- You know the approximate per-request cost of your deployment
- You can explain to another engineer (or a PM) when Modal beats the alternatives and when it doesn't
- Your endpoint is ready to connect to the Unified Client in Module 8
The next capsule
We start with 02: Account and CLI setup, where you:
- Create your free account with the initial credits
- Install and authenticate the Modal CLI
- Verify that you can run a remote function from your machine
Before moving on, make sure you have:
- Python 3.10+ installed
piporuvworking- An email account to register with Modal
- Basic terminal familiarity (it was covered if you come from Module 4 with Ollama)
Estimated time to finish the module: 2-3 hours.
Resources for the module
- Modal Docs — Getting Started — the official reference, the source of truth for syntax and pricing.
- Modal Pricing — current GPU/CPU/storage prices. They change; check before calculating production cost.
- Modal Examples Repo — official examples, including several LLMs.
- vLLM Documentation — the inference server we'll use on top of Modal to serve Mistral.
- "GPU serverless landscape 2024-2026" — context comparing Modal vs Replicate, RunPod, Beam (stay critical: Modal wrote it).
- Hugging Face Mistral 7B Instruct — the model we'll deploy.