1. Introduction: From Notebook to Production

Overview

What changes when your prompts serve real users: latency SLAs, cost budgets, reliability requirements, versioning, monitoring. From frictionless experimentation to production with accountability.

The Jump from Notebook to Production

There's a moment in every LLM project when you go from "this works on my machine" to "this needs to work for my users." That jump is bigger than it looks.

NOTEBOOK:
─────────
prompt = "Classify this: {text}"
response = client.chat.completions.create(...)
print(response.choices[0].message.content)
# It works ✓

PRODUCTION (what you actually need):
────────────────────────────────────
- What happens if the OpenAI API goes down?
- What happens if the output has the wrong format?
- How do I know if quality is degrading?
- How much will it cost me per month with 10,000 users?
- Can I roll back if a prompt change breaks something?
- How do I know if latency is blowing past the 3-second SLA?

What Changes in Production

The 5 Pillars of Production

This module covers the 5 pillars that make an LLM system production-ready:

1. Versioning and Registries

Without versioning, every prompt change is irreversible and opaque:

Before: "I changed the prompt to improve X, now it's worse but I don't know how to get back"
After: prompt_registry.rollback("classifier", to_version="v1.2")

2. Cost Optimization

Without optimization, cost grows non-linearly with users:

# Without optimization:
# 1,000 users/day × 500 tokens/request × $0.60/1M tokens = $0.30/day

# With optimization (compressed prompt + caching + routing):
# Same traffic → $0.08/day (73% reduction)

3. Caching

Without a cache, every request calls the API — even identical or very similar queries:

# Without cache: 1,000 requests of "What are your business hours?" → $1.50/day
# With cache (TTL 24h): same requests → $0.05/day on the second day

4. Prompt Management

Without a management system, prompts live in the code, in unstructured files, or in the developer's head:

Without management: the prompt is hardcoded in 3 different files
With management: prompt_manager.get("classifier", version="latest")

5. Monitoring and Observability

Without monitoring, you're the last to know when something breaks:

Without monitoring: "Hey, the classifier has been answering garbage for 2 days"
With monitoring: Alert within 5 minutes → "Accuracy dropped from 94% to 67%"

The Cost of Not Preparing

Let's look at the real numbers of not preparing for production:

# Scenario: Sentiment classifier, 10,000 requests/day

# Without cost optimization:
requests_per_day = 10_000
avg_tokens = 800  # Long, unoptimized prompt
input_price = 0.15 / 1_000_000   # GPT-4o-mini
output_price = 0.60 / 1_000_000

daily_cost = requests_per_day * avg_tokens * input_price
monthly_cost_no_opt = daily_cost * 30
print(f"Without optimization: ${monthly_cost_no_opt:.2f}/month")

# With optimization (200-token prompt + 40% cache hit rate):
optimized_tokens = 200
cache_hit_rate = 0.40
effective_requests = requests_per_day * (1 - cache_hit_rate)
cost_with_opt = effective_requests * optimized_tokens * input_price * 30
print(f"With optimization: ${cost_with_opt:.2f}/month")
print(f"Savings: ${monthly_cost_no_opt - cost_with_opt:.2f}/month")

Architecture of a Production-Ready LLM System

CLIENT
   │
   ▼
┌──────────────────────────────────────────────┐
│              API GATEWAY                      │
│  Rate limiting, Auth, Input validation        │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│           CACHE LAYER                         │
│  Exact match + Semantic cache                 │
│  Cache hit? → Return immediately              │
└──────────────────┬───────────────────────────┘
                   │ Cache miss
                   ▼
┌──────────────────────────────────────────────┐
│         PROMPT MANAGER                        │
│  Get prompt by name + version                 │
│  Render with variables                        │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│          MODEL ROUTER                         │
│  Classify complexity → route to right model   │
│  gpt-4o-mini (70%) vs gpt-4o (30%)            │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│         LLM API (OpenAI)                      │
│  With retry logic, timeout, error handling    │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│       OUTPUT VALIDATOR                        │
│  Format check, safety check                  │
│  Retry if invalid, fallback if needed         │
└──────────────────┬───────────────────────────┘
                   │
                   ▼
┌──────────────────────────────────────────────┐
│         MONITORING                            │
│  Log latency, cost, tokens, quality sample    │
│  Emit metrics to dashboard                    │
└──────────────────────────────────────────────┘

Module 8 Roadmap

The Difference Between Optimization and Negligence

A warning before we start: premature optimization is the enemy. But there's a difference between premature optimization and production negligence:

NEGLIGENCE (avoid):
✗ No error handling on API calls
✗ No token limit (risk of infinite costs)
✗ No prompt versioning
✗ No basic monitoring

PRODUCTION MINIMUM VIABLE (always have):
✓ Error handling with retries
✓ max_tokens configured
✓ Environment variables for API keys
✓ Basic error logging
✓ A way to roll back

ADVANCED OPTIMIZATION (when the system scales):
→ Semantic caching
→ Model routing
→ Prompt compression
→ Full observability stack

Module Reading Checklist

Before deploying any LLM system to production, you should be able to answer:

• Do you have versioning for your prompts with a way to roll back?
• Do you know the estimated cost per request and per month?
• Do you have caching for repeated queries?
• Do you have a system to manage and update prompts?
• Do you have monitoring for latency, errors and quality?
• Did you complete the production checklist before deploying?

If you answer "no" to any of them, this module gives you the tools to fix it.

Exercises

Exercise 1: Audit your current system

If you have an LLM system in production or in development, run this audit:

1. How much does it cost per month at current and projected traffic?
2. Can you roll back to a previous prompt version in < 5 minutes?
3. Do you have alerts when latency spikes or quality drops?

# Simple audit tool
def audit_llm_system(
    requests_per_day: int,
    avg_tokens_per_request: int,
    has_versioning: bool,
    has_caching: bool,
    has_monitoring: bool,
    has_error_handling: bool
) -> dict:
    """Generates a production readiness score."""
    
    # Calculate estimated monthly cost
    input_price = 0.15 / 1_000_000  # GPT-4o-mini
    daily_cost = requests_per_day * avg_tokens_per_request * input_price
    monthly_cost = daily_cost * 30
    
    # Readiness score
    checks = {
        "versioning": has_versioning,
        "caching": has_caching,
        "monitoring": has_monitoring,
        "error_handling": has_error_handling
    }
    
    score = sum(checks.values()) / len(checks) * 100
    
    return {
        "estimated_monthly_cost": f"${monthly_cost:.2f}",
        "readiness_score": f"{score:.0f}/100",
        "checks": checks,
        "recommendations": [k for k, v in checks.items() if not v]
    }

# Example:
result = audit_llm_system(
    requests_per_day=5000,
    avg_tokens_per_request=600,
    has_versioning=True,
    has_caching=False,  # Missing
    has_monitoring=False,  # Missing
    has_error_handling=True
)
print(result)

Exercise 2: Estimate the cost of your system

Given the following usage profile, estimate the monthly cost and the savings potential with caching:

• 50,000 requests/day
• 400 tokens per request on average
• 30% of queries are repeats (cache potential)

def estimate_cost_with_caching(
    requests_per_day: int,
    avg_tokens: int,
    cache_hit_rate: float,
    input_price_per_million: float = 0.15
) -> dict:
    price_per_token = input_price_per_million / 1_000_000
    
    # Without cache
    cost_without_cache = requests_per_day * avg_tokens * price_per_token * 30
    
    # With cache
    effective_requests = requests_per_day * (1 - cache_hit_rate)
    cost_with_cache = effective_requests * avg_tokens * price_per_token * 30
    
    savings = cost_without_cache - cost_with_cache
    
    return {
        "monthly_cost_without_cache": f"${cost_without_cache:.2f}",
        "monthly_cost_with_cache": f"${cost_with_cache:.2f}",
        "monthly_savings": f"${savings:.2f}",
        "savings_percentage": f"{savings/cost_without_cache*100:.0f}%"
    }

result = estimate_cost_with_caching(50_000, 400, 0.30)
print(result)
# {'monthly_cost_without_cache': '$90.00',
#  'monthly_cost_with_cache': '$63.00',
#  'monthly_savings': '$27.00',
#  'savings_percentage': '30%'}

Summary

• Production takes more than working code: SLAs, costs, reliability, versioning, monitoring
• The 5 pillars: Versioning, Cost, Caching, Management, Monitoring
• Full architecture: Cache → Prompt Manager → Router → LLM → Validator → Monitor
• Minimum viable: Error handling, max_tokens, secrets in env, basic logging, rollback possible
• This module: Concrete tools for each pillar, with runnable code

Module Prerequisites

Before continuing, make sure you're comfortable with:

• Python: Functions, classes, decorators, exception handling
• OpenAI SDK: Chat completions, structured output, temperature and max_tokens
• Pydantic: Input and output validation
• Environment variables: os.getenv(), .env files with python-dotenv
• Basic Git: Commits, branches, tags — needed for prompt versioning
• JSON/YAML: For configuration and prompt storage

If you completed Modules 1-7, you have everything you need. This module takes the techniques you learned and adds the layer of robustness they need to serve real users.

Frequently Asked Questions

Do I need a framework like LangChain for production? Not necessarily. Plain Python + the OpenAI SDK + good engineering practices take you a long way. Frameworks help with complex orchestration, but they add dependencies and abstractions. This module teaches you the fundamentals that apply with or without a framework.

What's the most common mistake when moving to production? Not having a cost plan. A prompt that costs $0.001 per call looks insignificant, but at 100,000 requests/day, that's $3,000/month. Capsule 03 gives you the tools to estimate, optimize and control costs.

How long does it take to put an LLM system in production? If you already have the prompt working, the 5 pillars of this module can be implemented in 1-2 weeks. The minimum viable setup (error handling + max_tokens + secrets + logging + rollback) can be done in a day.

Do I need special infrastructure? To start, no. An API (FastAPI/Flask) on any hosting service is enough. When you scale, you'll need caching (Redis), job queues (Celery/RQ), and monitoring (Prometheus/Grafana or services like Datadog). Capsules 04 and 06 cover this progressively.

Additional resources

1. OpenAI Production Best Practices — OpenAI's official guide
2. Anthropic Production — Anthropic's best practices
3. LLMOps: The Guide — MLOps community
4. Building LLM Applications for Production — Chip Huyen
5. The Production LLM Checklist — Eugene Yan