Cost optimization

Your endpoint works. You already know how to do autoscaling. Now comes the question they'll ask you when you show this in a meeting: "How much does each request cost us? And if we serve 1M a month?"

This capsule teaches you to answer that with numbers and to apply the optimizations with the most impact. It's not theory — by the end you'll have a mental spreadsheet of how much each deployment costs you and you'll be able to explain to your cofounder where you can cut 50% without sacrificing product.

By the end you'll be able to:

• Calculate the real per-request cost of your deployment (GPU + CPU + storage + egress)
• Choose the right GPU based on your model and expected throughput
• Apply the three highest-impact optimizations: batching, quantization, GPU sizing
• Decide when Modal stops being cheap and you should switch to an alternative

---

The mental model: three cost components

Your Modal bill breaks down into:

Component	When it's charged	Order of magnitude
GPU time	Per second, while a GPU container is on	$$$$ (the bulk)
CPU time	Per second, for containers without GPU	$
Storage	Per GB-month, for all volumes and images	$
Egress	Per GB transferred out of Modal	$ (negligible for text chat)

95% of the cost of an LLM deployment = GPU time. That's why this capsule focuses almost exclusively on optimizing GPU.

---

Calculating cost-per-request: the method

Three numbers give you the cost:

cost_per_request ≈ gpu_time_per_request × cost_per_second_of_GPU

Step 1 — Measure how much GPU your request uses

Modal's logs tell you how long each request takes on GPU. For an average request of your Mistral:

modal app logs mistral-api --tail 200 | grep "handled request"
# [container-abc123] handled request in 1.82s
# [container-def456] handled request in 2.04s
# [container-ghi789] handled request in 1.91s

Approximate average: ~2s of GPU per request.

Step 2 — Look at the per-second price of your GPU

From modal.com/pricing — as of early 2026, approximate prices (check the current one):

GPU	Approx cost per second
T4	~$0.00017
A10G	~$0.00030
L4	~$0.00025
A100 40GB	~$0.00100
H100	~$0.00250

With A10G and 2s per request:

cost_per_request = 2 × $0.00030 = $0.0006

Step 3 — Scale to your volume

Volume	Monthly GPU cost
1,000 requests	$0.60
100,000 requests	$60
1,000,000 requests	$600
10,000,000 requests	$6,000

Plus warm pool: if min_containers=1 keeps an A10G on 24/7, add ~$720/month regardless of volume.

---

Comparison with alternatives at this point

It's useful to have a reference. For 1M requests/month (~2 req/s sustained, 100 output tokens on average):

Option	Approx cost/month	Notes
Modal (Mistral 7B + warm pool)	~$600 + $720 = $1,320	Your model, your pricing
OpenAI GPT-4o-mini	~$300-500	Token-based; depends on length
Self-hosted Ollama (A10G on AWS 24/7)	~$1,800	GPU always on
OpenRouter (variable)	~$200-1,500	Depending on the model chosen

Modal with a warm pool doesn't always win. If your volume is high and constant, OpenAI can be cheaper. If you have privacy or model restrictions, Modal is still justified even if it costs a bit more.

---

The four highest-impact optimizations

Optimization 1 — Batching (gain: 2-4×)

vLLM automatically batches concurrent requests into a single GPU pass. Your client can send multiple prompts together:

@modal.method()
def generate_batch(self, prompts: list[str], max_tokens: int = 256) -> list[str]:
    from vllm import SamplingParams
    sampling = SamplingParams(temperature=0.7, max_tokens=max_tokens)
    formatted_prompts = [f"[INST] {p} [/INST]" for p in prompts]
    outputs = self.llm.generate(formatted_prompts, sampling)
    return [o.outputs[0].text.strip() for o in outputs]

Measurement: processing 8 prompts one by one = 8 × 2s = 16s. Processing the 8 in a batch = ~5s. Cost per request drops ~3×.

Applicability: it works if your use allows it (batch processes, embedded in async pipelines). It doesn't work for a 1-on-1 chatbot where each user waits for their response.

Optimization 2 — Quantization (gain: 2-3× in throughput, ~10% quality loss)

Mistral 7B in fp16 takes 14GB and processes ~50 tokens/s on A10G. The quantized AWQ (4-bit) version takes ~4GB and processes ~150 tokens/s. This means:

• Your request completes 2-3× faster → less GPU time → less cost
• It fits in smaller GPUs (T4 works for AWQ) → lower cost per second

MODEL = "TheBloke/Mistral-7B-Instruct-v0.2-AWQ"  # quantized version

@app.cls(
    image=image.pip_install("autoawq"),
    gpu="T4",  # it fits in T4 now!
    ...
)
class MistralServiceAWQ:
    @modal.enter()
    def load(self):
        from vllm import LLM
        self.llm = LLM(model=MODEL, quantization="awq")

Trade-off: quantization loses a bit of precision. For general conversational tasks, almost imperceptible. For tasks that require precise reasoning (math, code), evaluate before adopting.

Optimization 3 — Correct GPU sizing (gain: 30-60% from a GPU change)

More expensive isn't always = better for your case. The important question: are you "GPU-bound" or "memory-bound"?

If you're...	Symptom	Action
Memory-bound	"Out of memory" or full KV cache	Move to a GPU with more VRAM (A10G→L4 doesn't help; A10G→A100 does)
Compute-bound	GPU at 100% utilization, high latency	Move to a faster GPU (A10G→L4 or A100)
Over-provisioned	GPU at 30% utilization, latency ok	Drop to a smaller GPU

Seeing utilization: modal app logs or Modal's dashboard show GPU metrics. If your A10G never goes above 40% utilization, an L4 or T4 (with quantization) works just the same and costs less.

Optimization 4 — Reduce max_tokens reasonably (gain: linear)

Your latency (and cost) is proportional to the number of tokens generated. Generating 256 tokens costs 2× what generating 128 costs. If your product doesn't need very long responses:

class ChatRequest(BaseModel):
    prompt: str
    max_tokens: int = Field(150, ge=1, le=512)  # default 150, not 512

Combine it with prompt engineering: ask the model to be concise ("Answer in 3 sentences"). It works surprisingly well.

---

Storage: the forgotten component

Volumes charge ~$0.10/GB/month. If you have:

• mistral-weights (14GB) → ~$1.40/month
• llama-weights (40GB) → ~$4/month
• 10 forgotten versions of old models → it can add up

modal volume list                # see your volumes
modal volume du mistral-weights  # exact size
modal volume delete xxx          # delete

It's not a ton, but it's worth auditing quarterly. It costs zero to delete an old volume.

---

When Modal stops being cheap

Three signals that Modal is no longer the right path:

Signal 1 — Very constant and very high traffic (>10 req/s 24/7). If your GPU is saturated all the time, "pay per usage" ends up being more expensive than "flat rent". A reserved A10G on AWS costs ~$0.40/hr (vs $1+/hr on Modal). If your utilization is 95%, self-hosted wins.

Signal 2 — You need very specific GPUs that Modal doesn't offer. H200, MI300 (AMD), TPUs — Modal doesn't have them. If your model requires one, move to Replicate, RunPod, or a direct cloud.

Signal 3 — Strict compliance that Modal doesn't certify. HIPAA, specific SOC2, mandatory on-prem — check current certifications. Modal SOC2 Type II exists, but HIPAA may be a conversation with the sales team.

For 80% of cases at startups, Modal is optimal. The signals above are exceptions.

---

Common traps

Trap 1 — "The warm pool costs me more than the savings." Calculate honestly: if your min_containers=1 costs $720/month and you only avoid 100 cold starts a month, it's not worth it (each cold start costs ~$0.05). The warm pool is only worth it if the improved latency generates real value (real users who would leave if they wait 30s).

Trap 2 — "I have a free account, why am I being charged?" Modal's free tier is $30/month in credits. If you exceed it, you're charged the difference. Check your usage in the dashboard before the end of the month. Set billing alerts.

Trap 3 — "Quantization seemed free but my quality dropped a lot." Measure before adopting. Have a set of 20-50 representative prompts. Generate responses with fp16 and with AWQ. Compare manually or with an LLM-as-judge. If the quality difference matters for your case, don't adopt it.

Trap 4 — "I'm paying for GPU while the model downloads." Yes. The initial 14GB download is ~2-4 min of paid GPU. Optimize it: pre-bake weights into the image (we saw this in capsule 06) or use hf_transfer to accelerate 3-5×.

Trap 5 — "My traffic is bursty but max_containers=20 gives me cost spikes." Three options: (a) lower max_containers and tolerate queueing during bursts; (b) implement rate limiting in front; (c) add a cache of frequent responses (LangCache, semantic cache) so repeated requests don't touch the GPU.

---

Exercise: estimate your monthly cost

Your product has this profile:

• 50,000 requests/day
• Average 150 output tokens
• Mistral 7B on A10G
• 90% of traffic during business hours (9am-6pm)
• You require a warm pool during business hours

Estimate:

1. Active GPU cost (while generating responses)
2. Warm pool cost during business hours
3. Approximate total monthly cost
4. If you decide to quantize to AWQ and drop to T4, what would the new approximate total be?

See solution

Assuming: A10G at $0.00030/s, T4 at $0.00017/s, ~2s/request on A10G fp16, ~0.7s/request on T4 AWQ.

1. Active GPU cost (A10G fp16):

• 50,000 req/day × 30 days = 1,500,000 req/month
• 1,500,000 × 2s × $0.00030 = $900/month

2. Warm pool cost (business hours):

• 9hrs/day × 22 business days/month ≈ 200hrs/month
• 200hrs × 3600s × $0.00030 = $216/month

3. Approximate total: $900 + $216 + storage(~$2) = ~$1,120/month

4. With AWQ on T4:

• 1,500,000 × 0.7s × $0.00017 = $179/month (active GPU)
• 200hrs × 3600s × $0.00017 = $122/month (warm pool T4)
• Total: $179 + $122 + $2 = ~$303/month

Savings: ~73%. It's worth measuring whether the quality stays acceptable for your use case.

---

Summary

You learned:

• ✅ Break down your bill: GPU (the bulk) + CPU + storage + egress
• ✅ Calculate cost-per-request with two numbers: GPU time × price per second
• ✅ Four big optimizations: batching, quantization, GPU sizing, max_tokens
• ✅ When Modal stops being optimal: high constant traffic, specific GPUs, strict compliance
• ✅ Make a defensible estimate for a meeting with a cofounder/PM

Checkpoint: if they ask you "how much does your deployment cost a month with X traffic?" and you can give a number with a calculation, not just "cheap" or "I don't know", you're ready.

---

Next capsule

In 08 — Project: scalable API you're going to integrate everything from the module into a production-ready endpoint: with autoscaling configured, cost optimization applied, basic monitoring, and testing. It's the deliverable you'll use in the final project of Module 8 (Unified Client) and, hopefully, in something real of your own.

---

Resources

1. Modal Pricing — current prices per GPU.
2. vLLM Quantization Guide — AWQ step by step.
3. Modal — Billing dashboard — your current usage.
4. TheBloke on Hugging Face — collection of quantized models ready for vLLM.
5. Semantic caching for LLMs — reduce duplicate requests that touch the GPU.