7. Performance Baseline — Establishing Reference Metrics

Description

In this capsule you'll establish performance baselines for your deployed AI system: inference latency, monthly costs, error rates, and uptime. A baseline is your system's "normal" — without it, you don't know whether something is improving, degrading, or broken. By the end, you'll have documented targets and free tools to monitor them.

Context: Your system is deployed and validated. Now you need to define what "normal" is so you can detect anomalies. This is the bridge between this module (deployment) and guide #18 (Monitoring & Observability): here you establish the baselines that guide #18 teaches you to monitor with advanced tools.

Why You Need Baselines

Without a baseline vs with a baseline

WITHOUT A BASELINE:
- "The app seems slow" → Compared to what?
- "Costs went up" → Since when? How much is normal?
- "There are lots of errors" → What is "lots"?

WITH A BASELINE:
- "p95 latency is at 4.2s, the baseline is 2.5s" → 68% degraded, investigate
- "Costs this month: $45, baseline: $20" → 125% increase, check traffic
- "Error rate: 3.5%, baseline: 0.5%" → 7x increase, something is broken

Baselines turn subjective perceptions ("seems slow") into objective data ("it's 68% slower than normal").

The 4 essential metrics

┌─────────────────────────────────────────────────────────┐
│                 PERFORMANCE BASELINES                     │
│                                                          │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌─────────┐ │
│  │ LATENCY  │  │  COSTS   │  │  ERRORS  │  │ UPTIME  │ │
│  │          │  │          │  │          │  │         │ │
│  │ p50, p95 │  │ $/month  │  │ % rate   │  │ % avail.│ │
│  │ p99      │  │ breakdown│  │ by type  │  │ minutes │ │
│  └──────────┘  └──────────┘  └──────────┘  └─────────┘ │
│                                                          │
│  Target: <3s    Target:<$50  Target:<1%   Target:>99%    │
└─────────────────────────────────────────────────────────┘

Baseline 1: Latency

How to measure your AI system's latency

# scripts/measure_latency.py
"""
Measures the AI system's latency in production.
Run: python scripts/measure_latency.py https://your-app.railway.app 20
"""
import sys
import time
import json
import urllib.request
import statistics

def measure_endpoint(base_url: str, num_requests: int = 20):
    """Measures inference latency with multiple requests."""

    url = f"{base_url}/api/inference"
    latencies = []
    errors = 0

    print(f"Measuring latency: {num_requests} requests to {url}")
    print()

    for i in range(num_requests):
        payload = json.dumps({
            "prompt": f"Respond briefly: what is the number {i+1}?"
        }).encode()

        headers = {"Content-Type": "application/json"}
        req = urllib.request.Request(url, data=payload, headers=headers)

        start = time.time()
        try:
            response = urllib.request.urlopen(req, timeout=30)
            elapsed_ms = (time.time() - start) * 1000
            latencies.append(elapsed_ms)

            status = "OK" if response.status == 200 else f"HTTP {response.status}"
            print(f"  Request {i+1:3d}: {elapsed_ms:7.0f}ms — {status}")
        except Exception as e:
            elapsed_ms = (time.time() - start) * 1000
            errors += 1
            print(f"  Request {i+1:3d}: {elapsed_ms:7.0f}ms — ERROR: {e}")

        time.sleep(0.5)

    if not latencies:
        print("No successful requests")
        return

    latencies.sort()
    n = len(latencies)

    results = {
        "total_requests": num_requests,
        "successful": n,
        "errors": errors,
        "error_rate": f"{(errors/num_requests)*100:.1f}%",
        "latency_ms": {
            "min": round(latencies[0]),
            "max": round(latencies[-1]),
            "mean": round(statistics.mean(latencies)),
            "median_p50": round(latencies[n // 2]),
            "p90": round(latencies[int(n * 0.9)]),
            "p95": round(latencies[int(n * 0.95)]),
            "p99": round(latencies[int(n * 0.99)]) if n >= 100 else "N/A (need 100+ samples)",
            "std_dev": round(statistics.stdev(latencies)) if n > 1 else 0,
        },
    }

    print()
    print("=" * 50)
    print("LATENCY BASELINE RESULTS")
    print("=" * 50)
    print(f"  Requests: {n} successful / {num_requests} total")
    print(f"  Error rate: {results['error_rate']}")
    print(f"  Latency (ms):")
    print(f"    Min:    {results['latency_ms']['min']}ms")
    print(f"    p50:    {results['latency_ms']['median_p50']}ms")
    print(f"    p90:    {results['latency_ms']['p90']}ms")
    print(f"    p95:    {results['latency_ms']['p95']}ms")
    print(f"    Max:    {results['latency_ms']['max']}ms")
    print(f"    StdDev: {results['latency_ms']['std_dev']}ms")
    print()

    p95 = results["latency_ms"]["p95"]
    if p95 < 3000:
        print(f"  STATUS: GOOD — p95 ({p95}ms) < 3000ms target")
    elif p95 < 5000:
        print(f"  STATUS: WARNING — p95 ({p95}ms) > 3000ms but < 5000ms")
    else:
        print(f"  STATUS: CRITICAL — p95 ({p95}ms) > 5000ms")

    return results

if __name__ == "__main__":
    base_url = sys.argv[1] if len(sys.argv) > 1 else "http://localhost:8000"
    num_req = int(sys.argv[2]) if len(sys.argv) > 2 else 20
    measure_endpoint(base_url, num_req)

Expected output

Measuring latency: 20 requests to https://my-app.railway.app/api/inference

  Request   1:    3245ms — OK
  Request   2:    1890ms — OK
  Request   3:    2100ms — OK
  ...
  Request  20:    1945ms — OK

==================================================
LATENCY BASELINE RESULTS
==================================================
  Requests: 20 successful / 20 total
  Error rate: 0.0%
  Latency (ms):
    Min:    1654ms
    p50:    2100ms
    p90:    2890ms
    p95:    3100ms
    Max:    3245ms
    StdDev: 420ms

  STATUS: WARNING — p95 (3100ms) > 3000ms but < 5000ms

Define latency targets

latency_targets = {
    "health_check": {
        "target_ms": 200,
        "acceptable_ms": 500,
        "critical_ms": 1000,
    },
    "inference": {
        "target_ms": 2000,
        "acceptable_ms": 3000,
        "critical_ms": 5000,
    },
    "notes": [
        "Inference includes: network + preprocessing + LLM API call + postprocessing",
        "Most of the time (~70-80%) is the LLM call",
        "Cold starts on free tier can add 5-15s to the first request",
        "p95 is the main metric — not p50 nor the mean",
    ],
}

Baseline 2: Costs

Monthly cost breakdown

# scripts/cost_baseline.py
"""Calculates and documents the cost baseline."""

def calculate_monthly_costs(
    daily_requests: int,
    avg_input_tokens: int = 500,
    avg_output_tokens: int = 300,
    platform_cost: float = 0,
) -> dict:
    """Estimates monthly costs for an AI system."""
    monthly_requests = daily_requests * 30

    gpt4o_mini_input = 0.15 / 1_000_000
    gpt4o_mini_output = 0.60 / 1_000_000

    llm_input_cost = monthly_requests * avg_input_tokens * gpt4o_mini_input
    llm_output_cost = monthly_requests * avg_output_tokens * gpt4o_mini_output
    llm_total = llm_input_cost + llm_output_cost

    embedding_cost_per_request = 0.02 / 1_000_000 * 500
    embedding_total = monthly_requests * embedding_cost_per_request

    total = llm_total + embedding_total + platform_cost

    return {
        "monthly_requests": monthly_requests,
        "llm_cost": round(llm_total, 2),
        "embedding_cost": round(embedding_total, 2),
        "platform_cost": platform_cost,
        "total_monthly": round(total, 2),
        "cost_per_request": round(total / monthly_requests * 1000, 4),
    }

# Scenarios
scenarios = {
    "Current (150 users)": calculate_monthly_costs(
        daily_requests=500,
        avg_input_tokens=600,
        avg_output_tokens=300,
        platform_cost=0,  # free tier
    ),
    "Growth (500 users)": calculate_monthly_costs(
        daily_requests=2000,
        avg_input_tokens=600,
        avg_output_tokens=300,
        platform_cost=5,  # Railway Pro
    ),
    "Scale (2000 users)": calculate_monthly_costs(
        daily_requests=8000,
        avg_input_tokens=600,
        avg_output_tokens=300,
        platform_cost=20,  # Railway Pro + more resources
    ),
}

print("COST BASELINE")
print("=" * 60)
for name, costs in scenarios.items():
    print(f"\n{name}:")
    print(f"  Monthly requests: {costs['monthly_requests']:,}")
    print(f"  LLM cost:        ${costs['llm_cost']:.2f}/month")
    print(f"  Embedding cost:  ${costs['embedding_cost']:.2f}/month")
    print(f"  Platform cost:   ${costs['platform_cost']:.2f}/month")
    print(f"  TOTAL:           ${costs['total_monthly']:.2f}/month")
    print(f"  Per 1K requests: ${costs['cost_per_request']:.4f}")

Define cost targets

cost_targets = {
    "monthly_budget": 50,  # $50/month max
    "alert_threshold": 40,  # Alert at 80% of the budget
    "breakdown_expected": {
        "llm_api": "60-70% of the total",
        "platform": "20-30% of the total",
        "other": "<10% of the total",
    },
    "monitoring": {
        "openai": "platform.openai.com → Usage → set spending limit",
        "platform": "Dashboard → Billing → set alerts",
        "review": "Weekly: verify that spending is in line with the projection",
    },
}

Configure cost alerts

OpenAI:
  1. platform.openai.com → Settings → Limits
  2. Set monthly budget: $30 (leaves room for infra)
  3. Set email alert at: $20

Railway:
  1. Dashboard → Settings → Usage Alerts
  2. Or monitor the Usage meter manually

Fly.io:
  1. Dashboard → Billing → set spending limit

AWS:
  1. AWS Budgets → Create budget → Monthly cost budget
  2. Alert at 80% and 100% of budget

Baseline 3: Error Rate

Measure the current error rate

# scripts/error_baseline.py
"""Measures the system's error rate with varied requests."""
import sys
import json
import urllib.request
import urllib.error

def measure_error_rate(base_url: str, num_requests: int = 50):
    """Sends varied requests and measures the error rate."""

    test_cases = [
        {"prompt": "What is machine learning?", "expect": 200},
        {"prompt": "Explain Docker in one sentence", "expect": 200},
        {"prompt": "What is 2+2?", "expect": 200},
        {"prompt": "Summarize the benefits of CI/CD", "expect": 200},
        {"prompt": "Hello", "expect": 200},
    ]

    results = {"total": 0, "success": 0, "client_error": 0,
               "server_error": 0, "timeout": 0, "other": 0}
    error_details = []

    print(f"Measuring error rate: {num_requests} requests to {base_url}")

    for i in range(num_requests):
        test = test_cases[i % len(test_cases)]
        url = f"{base_url}/api/inference"
        payload = json.dumps({"prompt": test["prompt"]}).encode()
        headers = {"Content-Type": "application/json"}
        req = urllib.request.Request(url, data=payload, headers=headers)

        results["total"] += 1
        try:
            response = urllib.request.urlopen(req, timeout=30)
            if response.status == test["expect"]:
                results["success"] += 1
            else:
                results["client_error"] += 1
                error_details.append(f"Request {i+1}: expected {test['expect']}, got {response.status}")
        except urllib.error.HTTPError as e:
            if 400 <= e.code < 500:
                results["client_error"] += 1
            else:
                results["server_error"] += 1
            error_details.append(f"Request {i+1}: HTTP {e.code}")
        except TimeoutError:
            results["timeout"] += 1
            error_details.append(f"Request {i+1}: timeout")
        except Exception as e:
            results["other"] += 1
            error_details.append(f"Request {i+1}: {type(e).__name__}")

    total = results["total"]
    error_count = total - results["success"]
    error_rate = (error_count / total) * 100 if total > 0 else 0

    print()
    print("ERROR RATE BASELINE")
    print("=" * 50)
    print(f"  Total requests: {total}")
    print(f"  Successful:     {results['success']} ({(results['success']/total)*100:.1f}%)")
    print(f"  Client errors:  {results['client_error']}")
    print(f"  Server errors:  {results['server_error']}")
    print(f"  Timeouts:       {results['timeout']}")
    print(f"  Other errors:   {results['other']}")
    print(f"  ERROR RATE:     {error_rate:.1f}%")
    print()

    if error_rate < 1:
        print(f"  STATUS: GOOD — error rate ({error_rate:.1f}%) < 1% target")
    elif error_rate < 5:
        print(f"  STATUS: WARNING — error rate ({error_rate:.1f}%) > 1% but < 5%")
    else:
        print(f"  STATUS: CRITICAL — error rate ({error_rate:.1f}%) > 5%")

    if error_details:
        print(f"\n  Error details ({len(error_details)}):")
        for detail in error_details[:10]:
            print(f"    - {detail}")

    return results

if __name__ == "__main__":
    url = sys.argv[1] if len(sys.argv) > 1 else "http://localhost:8000"
    num = int(sys.argv[2]) if len(sys.argv) > 2 else 50
    measure_error_rate(url, num)

Define error rate targets

error_targets = {
    "overall_error_rate": {
        "target": "< 1%",
        "acceptable": "< 5%",
        "critical": "> 5%",
    },
    "by_type": {
        "server_errors_5xx": "< 0.1% — indicates bugs in your code",
        "timeouts": "< 0.5% — indicates infra or LLM API problems",
        "client_errors_4xx": "Doesn't count as an 'error' — it's incorrect client usage",
    },
    "notes": [
        "The OpenAI API has ~0.1-0.5% of its own error rate (rate limits, server errors)",
        "Free tier platforms have more errors due to cold starts and resource limits",
        "Error rate = (server_errors + timeouts) / total_requests",
    ],
}

Baseline 4: Uptime

Measure and monitor uptime

Uptime = time the service is accessible / total time

Target    Downtime/month  Downtime/year
99%       7.3 hrs         3.65 days
99.5%     3.65 hrs        1.83 days
99.9%     43.8 min        8.77 hrs
99.99%    4.38 min        52.6 min

For an AI system on free tier:
  Realistic target: 99% (7 hrs downtime/month max)
  Includes: cold starts, platform maintenance, deploys

For an AI system on a paid plan:
  Realistic target: 99.5% (3.65 hrs downtime/month)

For enterprise production:
  Target: 99.9%+ (requires dedicated infra)

Free uptime monitoring tools

UptimeRobot (recommended):
  - 50 free monitors
  - Check every 5 minutes
  - Public status page (optional)
  - Alerts: email, Slack, webhook
  - Uptime history (monthly %)

Configuration:
  Monitor 1: GET /health → every 5 min
  Monitor 2: POST /api/inference with body → every 15 min (verifies inference)

Better Stack:
  - 10 free monitors
  - Check every 3 minutes
  - Status page included
  - Basic incident management

Freshping:
  - 50 free monitors
  - Check every 1 minute
  - Multi-location checks

Performance Baseline Document

Complete template

# Performance Baseline — [AI System Name]

**Baseline date:** [Date]
**Production URL:** [URL]
**Platform:** [Railway/Render/Fly.io/AWS]
**Plan:** [Free/Pro/etc.]

## Latency

| Metric | Baseline value | Target | Alert if |
|---------|---------------|--------|-----------|
| p50 | [X]ms | <2000ms | >2500ms |
| p90 | [X]ms | <2500ms | >3000ms |
| p95 | [X]ms | <3000ms | >4000ms |
| Max | [X]ms | <5000ms | >8000ms |
| Health check | [X]ms | <200ms | >500ms |

**Measured with:** [N] requests on [date]
**Note:** [Observations about cold starts, variability, etc.]

## Costs

| Component | Baseline | Budget | Alert if |
|-----------|----------|--------|-----------|
| LLM API (OpenAI) | $[X]/month | $[Y]/month | >$[Z]/month |
| Platform (infra) | $[X]/month | $[Y]/month | >$[Z]/month |
| Total | $[X]/month | $[Y]/month | >$[Z]/month |
| Per 1K requests | $[X] | - | >$[Y] |

**Spending alerts configured:** [Yes/No] [Where]

## Error Rate

| Metric | Baseline value | Target | Alert if |
|---------|---------------|--------|-----------|
| Overall error rate | [X]% | <1% | >2% |
| Server errors (5xx) | [X]% | <0.1% | >0.5% |
| Timeouts | [X]% | <0.5% | >1% |

**Measured with:** [N] requests on [date]

## Uptime

| Metric | Target | Monitoring |
|---------|--------|-----------|
| Monthly uptime | >99% | UptimeRobot |
| Max downtime/incident | <30 min | Alerts configured |
| Max downtime/month | <7 hrs | Monthly review |

**Monitoring configured:** [Tool, dashboard URL]
**Alerts configured:** [Email/Slack when downtime >5 min]

## When to Scale

| Trigger | Current metric | Threshold | Action |
|---------|---------------|-----------|--------|
| Sustained latency | [X]ms p95 | >5000ms for >1 hour | Scale resources or optimize |
| High error rate | [X]% | >5% for >30 min | Investigate + possible rollback |
| Costs exceeded | $[X]/month | >$[Y]/month | Traffic review and optimization |
| High memory | [X]% | >90% sustained | Scale RAM or optimize |

When to Scale

Signs that you need more resources

SCALE VERTICALLY (more CPU/RAM):
├── Sustained p95 latency > 2x baseline for > 1 hour
├── Memory > 90% of the available limit
├── CPU > 80% sustained
└── Action: Upgrade the plan on the platform

SCALE HORIZONTALLY (more instances):
├── Concurrent requests > one instance's capacity
├── Request queue growing
├── Latency increases linearly with traffic
└── Action: Add replicas (if the platform allows it)

OPTIMIZE CODE (before scaling infra):
├── Cache repetitive responses
├── Reduce tokens in prompts
├── Switch to a faster model (gpt-4o-mini vs gpt-4o)
├── Async processing for heavy requests
└── Action: Optimize BEFORE paying for more infra

Decision tree for scaling

High latency?
├── Is it the LLM API? (>70% of the total time)
│   ├── YES → Optimize prompts, reduce tokens, or change the model
│   └── NO → Is it your code?
│       ├── YES → Profiling, optimize, caching
│       └── NO → Is it the platform?
│           ├── YES → Scale resources or change platform
│           └── NO → Investigate networking
│
High costs?
├── Is it the LLM API? (check token usage)
│   ├── YES → Reduce tokens, caching, rate limiting
│   └── NO → Is it the infra?
│       ├── YES → Evaluate the current plan vs alternatives
│       └── NO → Legitimate traffic or abuse?
│           ├── Legitimate → Scale (invest in growth)
│           └── Abuse → Rate limiting, firewall

Troubleshooting

Problem 1: "Latency varies a lot between requests"

Cause: Cold starts, LLM API variability, or garbage collection in Python.

Solution:

# 1. Exclude the first request (cold start) from the baseline
latencies = latencies[1:]  # Skip the warm-up request

# 2. Report the standard deviation along with p50/p95
# High StdDev (>50% of the mean) = high variability
# Low StdDev (<20% of the mean) = consistent

# 3. Implement warm-up in the pipeline
# After the deploy, send 2-3 warm-up requests before the smoke tests

Problem 2: "I can't measure costs because I'm on free tier"

Cause: The free tier has no visible billing.

Solution: Measure usage, not the direct cost.

# Measure OpenAI usage (which does have billing)
# platform.openai.com → Usage → view by day

# Estimate the infra cost if you migrate to a paid plan
estimated_infra = {
    "Railway Pro": 5,        # $5/month base
    "Render Starter": 7,     # $7/month
    "Fly.io": 0,             # Free VMs, pay for additional usage
    "AWS Lambda": 0,         # Free tier: 1M requests
}

Problem 3: "The error rate is 0% but I know it sometimes fails"

Cause: You're measuring with too few requests or with very simple prompts.

Solution:

# Use more varied and realistic prompts
edge_case_prompts = [
    "A" * 5000,                    # Very long prompt
    "🎉 emoji test 中文 عربي",      # Unicode
    "Repeat the word 'test' 500 times",  # High-token request
    "",                             # Empty prompt (should give 422)
    "x" * 100000,                  # Extremely long prompt
]
# Measure with at least 50 requests for a meaningful baseline

Problem 4: "I don't know what targets to set for latency"

Solution:

The rule of thumb for AI apps:
- If it's interactive chat: p95 < 3s (users expect a fast response)
- If it's a backend API: p95 < 5s (depends on the consumer)
- If it's batch processing: p95 < 30s (it's not interactive)

Baseline first, targets later:
1. Measure 20+ requests
2. Calculate p50 and p95
3. Target = current p95 + 20% margin
4. Alert = current p95 + 50% margin
5. Critical = current p95 × 2

Hands-On Exercises

Exercise 1: Measure the latency baseline

Run the latency measurement script against your deployed system and document the results.

# Run the measurement
python scripts/measure_latency.py https://your-app.railway.app 20

# Expected result:
# p50: ~2000ms (depends on the model and platform)
# p95: ~3000ms
# StdDev: ~400-800ms

# Document in your performance baseline:
# "Latency measured on [date] with 20 requests.
#  p50: 2100ms, p95: 3050ms, max: 3500ms.
#  80% of the time is the call to the OpenAI API.
#  Cold start of the first request: 12s (excluded from the baseline)."

Exercise 2: Calculate the cost baseline

Run the cost script with your real data and configure alerts.

# With your data:
my_costs = calculate_monthly_costs(
    daily_requests=500,     # Your current traffic
    avg_input_tokens=600,   # Measure with tiktoken
    avg_output_tokens=300,  # Estimate or measure
    platform_cost=0,        # Free tier
)

print(f"Estimated monthly cost: ${my_costs['total_monthly']:.2f}")
# Example: $4.20/month on free tier with 500 req/day

# Configure an alert on OpenAI:
# platform.openai.com → Settings → Limits → $10/month

Exercise 3: Measure the error rate

Run the error rate script and document the results.

python scripts/error_baseline.py https://your-app.railway.app 50

# Expected result:
# Error rate: 0-2% (depends on your system's stability)
# Server errors: 0% (ideal)
# Timeouts: 0-1% (can happen on free tier)

# If error rate > 5%:
# 1. Review the error details
# 2. Are they all the same type? (e.g., all timeouts)
# 3. Is it the OpenAI API? (rate limiting)
# 4. Is it your code? (bug in the handler)

Exercise 4: Complete performance baseline document

Create docs/performance-baseline.md using this capsule's template with your real data.

# Performance Baseline — My AI System

**Baseline date:** 2026-03-08
**Production URL:** https://my-app.railway.app
**Platform:** Railway (Free tier)

## Latency
| Metric | Baseline | Target | Alert |
|---------|----------|--------|--------|
| p50 | 2100ms | <2500ms | >3000ms |
| p95 | 3050ms | <3500ms | >4500ms |

Measured with 20 requests on 2026-03-08.
Note: First request cold start ~12s (excluded).

## Costs
| Component | Baseline | Budget |
|-----------|----------|--------|
| OpenAI API | $4.20/month | $10/month |
| Railway | $0/month (free) | $5/month |
| Total | $4.20/month | $15/month |

## Error Rate
| Metric | Baseline | Target |
|---------|----------|--------|
| Overall | 0.0% | <1% |
| Timeouts | 0.0% | <0.5% |

Measured with 50 requests on 2026-03-08.

## Uptime
Monitoring: UptimeRobot (every 5 min)
Target: >99% monthly

Summary

• Baselines turn perceptions ("seems slow") into data ("p95 is 68% over the target")
• The 4 essential metrics: latency (p50/p95), costs ($/month), error rate (%), uptime (%)
• Measure before defining targets — targets are based on real data, not aspirations
• Free tools cover the basic monitoring: UptimeRobot, OpenAI Usage dashboard, your own scripts
• When to scale: first optimize code (caching, shorter prompts), then scale infra
• The performance baseline document is a living artifact — update it monthly
• This baseline is the input for guide #18 (Monitoring & Observability), which goes deeper into advanced metrics

Additional Resources

1. UptimeRobot — Free uptime monitoring
2. OpenAI Usage Dashboard — API usage monitoring
3. Google SRE Book — Service Level Objectives — SLOs and SLIs
4. Latency Numbers Every Programmer Should Know — Latency reference
5. tiktoken — OpenAI Tokenizer — Count tokens to estimate costs
6. Cloud Cost Handbook — Cloud cost reference