Capsule 07: Observability — what to measure when nobody debugs with prints anymore

Capsule description

In development, debugging a RAG is trivial: you run the script, read the prints, see which chunks it returned, compare with the LLM's answer. In production none of that applies. The system processes 10,000 queries per hour, each query generates 5-7 chunks, and the LLM answers different things depending on the context. If a query returns a bad answer at 3 AM on a Saturday, how do you find out? How do you investigate the cause? How do you distinguish a real bug from expected variance?

Observability is the answer. It's not "adding logs" — it's designing the system to be diagnosable from the start, continuously measuring the five dimensions that matter for RAG: latency, throughput, accuracy, cost, and index health. When something fails, observability tells you where it fails in less than 5 minutes instead of 5 hours. When the system works well, it tells you why and how to replicate that state.

This capsule is conceptual — the deep operational code comes in M07 (Production Considerations) and guide #18 (Monitoring & Observability). Here you build the mental model: what matters to measure, why, what thresholds trigger action, and how to design the metrics schema from day 1 so your future self doesn't hate you.

By the end of this capsule you'll be able to:

• ✅ List the five critical dimensions to monitor in RAG (latency, throughput, accuracy, cost, index)
• ✅ Differentiate between health metrics (the system works) and quality metrics (the system works well)
• ✅ Distinguish what to measure continuously vs what to measure periodically (golden set evaluation)
• ✅ Design alerting based on SLOs (Service Level Objectives) instead of arbitrary alerts
• ✅ Anticipate the most expensive trap: optimizing what you measure and breaking what you don't

Estimated time: 25-30 minutes

---

Why you need observability from day 1

Three situations that happen in RAG systems without observability:

Situation 1 — the slow query with no visible cause. Users complain that "the bot is slow". You open the code, you don't see anything odd. You test locally and it's fast. Is it the network? The LLM? The vector DB? Without instrumented metrics, it's 4 hours of manual bisection before discovering it was OpenAI's rate limit kicking in during peak hours.

Situation 2 — the bad answer the customer reports. A customer says "this answer is wrong, it said X when the documentation says Y". You open the system and... how do you reproduce the exact query from 3 days ago? What chunks did it return? What prompt did it use? Without traces, you can't even start to investigate.

Situation 3 — the slow degradation nobody notices. Your accuracy drops from 95% to 78% over 3 months. Nobody finds out because there's no continuous test. Users get used to worse answers. Until the competitor improves and you start losing clients — and only then do you realize you've had a degraded system for months.

Each of these is prevented with well-designed observability. The cost of instrumenting on day 1 is ~10% of the development time. The cost of instrumenting after an incident is 10x — plus the cost of the incident.

---

The five dimensions that matter in RAG

Dimension 1: Latency (how fast does it respond?)

Metrics to record for each query:

{
    "total_latency_ms": 245,           # End-to-end time
    "embedding_latency_ms": 110,       # Time to embed the query (OpenAI)
    "retrieval_latency_ms": 18,        # Search time in ChromaDB
    "generation_latency_ms": 117,      # Generation time with GPT
    "total_chunks_retrieved": 5,
}

Why break it down: "the system is slow" is useless information. "The embedding takes 110ms" tells you the bottleneck is OpenAI, not ChromaDB. The breakdown is what lets you fix the right problem.

Aggregate metrics to compute:

• p50_latency, p95_latency, p99_latency per component
• Distribution of latencies in a histogram (10ms, 50ms, 100ms, 200ms, 500ms, >500ms)

Typical SLOs for RAG production:

Component	Target p95	Target p99
Embedding (OpenAI)	<200ms	<400ms
Retrieval (ChromaDB)	<50ms	<150ms
Generation (GPT-4o-mini)	<800ms	<2000ms
Total	<1200ms	<3000ms

If your product tolerates more latency (async chatbot, internal bot), the targets are more relaxed. If it competes with search engines (Google ~200ms total), they're stricter.

Dimension 2: Throughput (how much work goes through the system?)

Metrics:

• QPS (queries per second): current and average load
• Concurrent users: how many active users at the same time
• Rate of fallbacks: % of queries where the LLM said "I don't have information" (a metric of dataset coverage)

Why it matters: it sizes your infrastructure. If your peak is 50 QPS and your system supports 100, all good. If your peak is 50 QPS and your system supports 60, a marketing promotion can take down the service.

Typical alert: if qps_current > 0.8 * qps_capacity: alert("approaching capacity"). You need to know your system's real qps_capacity, not the theoretical one — you discover that by load testing before production.

Dimension 3: Accuracy (how well does it respond?)

This is the most subtle one. Latency is trivial to measure; accuracy requires golden set evaluation.

How it works:

1. You build a golden set: 30-100 queries with answers annotated by humans as "correct" or "expected".
2. You run the golden set against the system periodically (daily, weekly).
3. Metrics it produces:
   • Recall@K: % of correct chunks in the top-K retrieved
   • MRR (Mean Reciprocal Rank): how high the first correct chunk ranks
   • NDCG@K: relevance-weighted ranking
   • Answer correctness: human or LLM-as-judge evaluation of the final answer

Common trap: measuring only retrieval accuracy, not answer correctness. The system can retrieve the correct chunks and still the LLM can answer wrong (hallucination, ignoring context). The two metrics measure different things.

# Pseudo-code for golden set evaluation
def evaluate_golden_set(rag_system, golden_set):
    metrics = {"retrieval_accuracy": 0, "answer_correctness": 0}
    for item in golden_set:
        response = rag_system.ask(item["query"])

        # Metric 1: did retrieval find the correct chunks?
        retrieved_ids = {s["doc_id"] for s in response.sources}
        relevant_ids = set(item["expected_doc_ids"])
        if retrieved_ids & relevant_ids:
            metrics["retrieval_accuracy"] += 1

        # Metric 2: is the answer correct?
        is_correct = check_answer(response.answer, item["expected_answer"])
        if is_correct:
            metrics["answer_correctness"] += 1

    n = len(golden_set)
    return {
        "retrieval_accuracy": metrics["retrieval_accuracy"] / n,
        "answer_correctness": metrics["answer_correctness"] / n,
    }

Recommended frequency: daily in production. If the metric drops >5% from the baseline, alert to investigate.

Going deeper: guide #12 (Evaluation Frameworks) covers this in detail, including Ragas, TruLens, and LLM-as-judge.

Dimension 4: Cost (how much does each query cost?)

{
    "embedding_cost_usd": 0.000010,    # OpenAI embedding of the query
    "generation_cost_usd": 0.000180,   # OpenAI GPT-4o-mini generation
    "total_cost_usd": 0.000190,
}

Aggregate metrics:

• Total monthly cost per component (embeddings, generation, vector DB hosting)
• Average cost per query
• Cost per active user

Why measure: without tracking, costs grow unchecked. A change in the prompt that adds 2K tokens of context can raise the monthly cost 30% without anyone noticing until the bill.

Common trap: measuring only the OpenAI generation cost (typically the highest) and ignoring embeddings and vector DB. When the dataset grows to millions, the cost of re-embedding (when you change models) and of hosting the vector DB can become significant.

Dimension 5: Index health (is the index OK?)

These are operational metrics of the backend:

{
    "index_size_bytes": 6_400_000_000,        # ~6 GB for 1M 1536-dim vectors
    "total_documents": 1_023_456,
    "ram_usage_pct": 78,                       # % of available RAM used
    "failed_inserts_last_hour": 12,
    "failed_queries_last_hour": 0,
    "last_index_rebuild": "2026-05-01T03:15:00Z",
    "ef_search_current": 50,
    "ef_construction_at_build": 200,
}

Alerts:

• failed_inserts > 100/hour: probably a rate limit or network problem
• ram_usage_pct > 90: OOM danger, scale up or migrate to IVF+PQ
• last_index_rebuild >30 days: consider a rebuild to maintain quality

---

Continuous vs periodic — what to measure when

Not all metrics need to be measured on every query. Some are too expensive or noisy at that rate.

Continuous metrics (every query)

• Latency (all components)
• Cost
• Number of chunks retrieved
• Whether there was a fallback (LLM said "I don't know")
• Errors (timeout, rate limit, etc.)

These are logged in an observability system (Datadog, Prometheus, OpenTelemetry) and processed in real time for dashboards.

Periodic metrics (golden set evaluation)

• Recall@K
• MRR
• Answer correctness

These require running against a fixed set of queries with known answers. They're run daily or weekly, not every query (that would be expensive and noisy).

Per-event metrics

• Drift detection: when the embeddings model or the prompt changes, compare before/after accuracy on the golden set
• Regression alerts: a PR that introduces a change to the system runs the golden set in CI; if accuracy drops >2%, it blocks the merge

---

Design alerting with SLOs, not arbitrary thresholds

The common error: defining alerts with values that "sound reasonable" without justification.

# ❌ Arbitrary thresholds
if p95 > 200:
    alert()
if accuracy < 0.85:
    alert()

Why 200ms and not 250ms? Why 0.85 and not 0.80? Without justification, you'll have alerts that fire without being a real problem (alert fatigue) or don't fire when it is one.

The right approach: SLOs (Service Level Objectives) based on what the product needs:

# ✅ SLO derived from product requirements
# "95% of queries must respond in <500ms" → SLO = p95 latency <500ms
# "Tolerable up to 1% error rate per hour" → SLO = error rate <1%
# "Accuracy must stay within 5% of the baseline" → SLO = accuracy >= baseline * 0.95

SLOs = {
    "p95_latency_ms": 500,
    "error_rate_pct": 1.0,
    "accuracy_relative_to_baseline": 0.95,
}

# Alert when you've been out of the SLO for more than 5 minutes
def check_slo(metric_name, value, threshold):
    if not within_slo(value, threshold):
        increment_breach_counter(metric_name)
        if breach_counter[metric_name] > 5:  # 5 consecutive minutes
            alert(severity="page_oncall", metric=metric_name, value=value)

Severity levels:

• Page (wake someone up): breach of a critical SLO (system down, accuracy <80% of the baseline)
• Ticket (review tomorrow): breach of a minor quality SLO (latency 10% over target, sustained)
• Log only (no immediate action): anomaly detected but within tolerance (1 query in 10K took 3s)

Golden rule: an alert that wakes someone up should have a clear action to resolve it. If the alert fires and nobody knows what to do, it's not a useful alert — it's noise.

---

The most expensive trap: optimizing what you measure and breaking what you don't

Goodhart's Law applied: "When a metric becomes a target, it stops being a good metric."

Concrete example: you measured latency and nothing else. The team aggressively optimizes p95 from 500ms to 200ms. How? They lower n_results from 10 to 3, and ef_search from 50 to 10. Latency improves, everything looks green on the dashboard.

What you weren't measuring: accuracy dropped from 92% to 76%. The system responds fast with worse answers. Users start complaining, but not in metrics — in support tickets that take you weeks to correlate with the change.

How to prevent it:

1. Measure the five dimensions always, not just the ones that seem important. A change that improves one metric can silently worsen another.
2. Have "protection metrics": when you optimize latency, define an SLO on accuracy that must NOT be broken. If it breaks, revert the change even if the latency improved.
3. Compound metrics: monitor ratios like "latency × (1 / accuracy)" — it gets worse if either of the two degrades.
4. A/B testing before deploying big changes — compare the two versions against the golden set + continuous metrics during 1-2 weeks before promoting.

---

Traps and common errors

Trap 1: measuring averages, ignoring percentiles

Error: the dashboard shows "average latency 80ms" → it looks good.

Reality: 90% of queries respond in 30ms, 10% in 500ms. The 80ms average hides that 1 in 10 users suffers.

How to prevent it: always p50, p95, p99 (covered in M04/06). The average is only reported as a complement.

Trap 2: golden set that ages

Error: golden set built 8 months ago with queries that no longer represent real usage. The reported accuracy stays at 95%, but the queries that actually arrive are failing.

How to prevent it: revisit the golden set every 3-6 months. Better yet: sample 20-50 real queries per month (anonymized), annotate them, and add them to a rotating golden set.

Trap 3: alerting without a runbook

Error: an alert fires "p95 high". The person on call doesn't know what to do. Calls the team. They take 2 hours to triage.

How to prevent it: each alert must have an associated runbook:

## Alert: p95_latency_ms > 500 for >5 min

**Steps:**
1. Check the embedding latency dashboard (https://...). If >300ms p95, problem with OpenAI → see runbook OpenAI-Outage.
2. Check current QPS vs capacity. If >85%, scale manually: `kubectl scale deployment rag-api --replicas=8`.
3. Check errors in the logs for the last 5 min: `kubectl logs ... | grep ERROR`. If there's a rate limit, temporarily increase the backoff.
4. If none of the above applies, roll back to the previous deploy: `./scripts/rollback.sh`. Post-mortem investigation tomorrow.

Trap 4: only technical metrics, no product metrics

Error: you monitor latency, technical accuracy, cost. But you don't monitor "% of users who make another query after the first" or "average answer rating" or "% of fallbacks".

Symptom: the engineers say "everything is green" while the product is in a retention decline.

How to prevent it: align with the product team. Define 2-3 user-impact metrics (retention, NPS, % of queries with a thumbs up) and monitor those too.

Trap 5: traces without enough context to reproduce

Error: you log "query: 'how do I configure X', response: 'no information found'". 3 days later you can't tell why it failed — what chunks were retrieved? What was the exact prompt? Which model version was it?

How to prevent it: log the full trace per query (with sampling in production so you don't fill up storage):

trace = {
    "trace_id": "uuid-...",
    "timestamp": "2026-05-08T...",
    "query": query_text,
    "query_embedding_model": "text-embedding-3-small",
    "retrieved_chunks": [{"doc_id": ..., "score": ..., "text_preview": ...}],
    "prompt_template_version": "v3",
    "llm_model": "gpt-4o-mini",
    "llm_temperature": 0,
    "response": response_text,
    "fallback": False,
    "user_feedback": None,  # filled in later if the user rates it
}

Sample 10% in production, 100% in staging. Cheap storage, high value when a bug shows up.

Trap 6: privacy violations in logs

Error: you log the user's full query. Some queries contain PII (names, emails, medical data). Unencrypted logs become a legal liability.

How to prevent it:

• Define what is logged and what is not according to privacy policies and regulation (GDPR, HIPAA if applicable).
• PII redaction before logging (regex over emails, IDs).
• Encryption at rest for logs.
• Retention policies (e.g., delete detailed traces after 30 days).

---

Applied exercise

Scenario: you joined a team where the RAG has been in production for 6 months without serious observability. The logs are print() in stdout. There's no golden set. There's a dashboard that shows "average latency = 120ms" and nothing else.

Reported symptoms:

• Some users say "the bot responds slowly", others say "it responds badly"
• Nobody knows if the system got worse or was always like this
• The product team asks "can we know which types of query fail the most?" and nobody has an answer

Your job: design an observability instrumentation plan for the next 30 days. Specify what to measure, how, in what order of priority, and what to report to the stakeholder at the end of the month.

Solution

Plan in 4 weekly sprints:

---

Week 1: basic latency and error instrumentation

Goal: stop operating blind. Know the most basic things by the end of the week.

Actions:

1. Add a tracing middleware (OpenTelemetry or a managed solution like Datadog) that captures for each query:

   • total_latency_ms, embedding_latency_ms, retrieval_latency_ms, generation_latency_ms
   • chunks_retrieved, fallback_triggered (bool)
   • error_type if there was an error
   • unique trace_id

2. Basic dashboard with:

   • p50/p95/p99 of each component
   • current and last-24h QPS
   • Error rate
   • Fallback rate distribution

3. Minimal alerts:

   • p95 total > 1500ms for >5 min → ticket
   • Error rate > 5% for >2 min → page

Stakeholder report (end of week 1): "Now we know how long the system really takes. Current p95 = 850ms, vs the 'average 120ms' we'd been seeing. The bottleneck is generation (450ms p95), not retrieval (50ms p95). Next step: measure how well it responds, not just how fast."

---

Week 2: golden set + accuracy baseline

Goal: establish a quality baseline to detect degradation.

Actions:

1. Build an initial golden set: 50 representative queries with answers annotated by the product team.
2. Automated evaluation pipeline that runs the golden set daily:
   • Recall@5 (were the correct chunks retrieved?)
   • Answer correctness (LLM-as-judge initially, then weekly human review)
3. Accuracy dashboard:
   • Accuracy trend over the last 30 days
   • Breakdown by query category
4. Alert:
   • Daily accuracy < baseline * 0.95 → ticket for investigation

Stakeholder report (end of week 2): "Accuracy baseline established: recall@5 = 87%, answer correctness = 84%. This is our comparison point from here on out. If it drops >5%, we'll know something changed."

---

Week 3: detailed traces and debugging

Goal: be able to investigate why a specific query failed.

Actions:

1. Trace logging with 10% sampling in production, 100% in staging:
   • Query text (with PII redaction)
   • Chunks retrieved with scores
   • Full prompt sent to the LLM
   • The LLM's response
   • Model, temperature, other parameters
2. Trace search by trace_id (Datadog APM, Honeycomb, etc.).
3. Admin endpoint to "investigate query" — you enter a trace_id and it gives you the full flow.

Bonus: targeted sampling — save 100% of traces when there's an error or when the user rating is negative.

Stakeholder report (end of week 3): "When a user reports a bad answer, we can now investigate in <10 minutes instead of 'we don't know what happened'. We identified 3 systematic error patterns: (a) queries in Spanish return irrelevant chunks in English, (b) very specific queries about new products have no documentation, (c) ambiguity of legal terms causes poor retrieval."

---

Week 4: SLOs, runbooks, product metrics

Goal: align with product and formalize operations.

Actions:

1. Define SLOs with the product stakeholders:
   • p95 latency < 1000ms
   • Error rate < 0.5%
   • Daily accuracy >= baseline_accuracy * 0.95
2. Runbooks for each critical alert (what to do when it fires).
3. Add product metrics:
   • User feedback (thumbs up/down) per answer
   • Rate of reformulated queries (a signal of a bad initial answer)
   • Weekly retention of users who used the bot
4. Automated monthly report for stakeholders.

Final stakeholder report (end of the month):

"In 30 days we went from operating blind to having full observability. Summary:

- Real measured latency: p95 = 850ms (not 120ms as we thought). We identified generation with OpenAI as the bottleneck; a clear optimization opportunity. - Accuracy baseline: recall@5 = 87%, answer correctness = 84%. Trend monitored daily. - 3 error patterns identified that we can now attack by priority. - SLOs defined and aligned with the product. Alerts with an actionable runbook. - Debugging capability: from 'we don't know what happened' to investigation in <10 min.

Next 30 days: attack the 3 identified error patterns (prioritizing the one with queries in Spanish that affects 22% of traffic) and work on lowering p95 latency from 850ms to 500ms."

---

Summary and next step

What you learned:

• Observability isn't adding logs — it's designing the system to be diagnosable from day 1.
• Five dimensions to measure in RAG: latency, throughput, accuracy, cost, index health.
• The difference between continuous metrics (every query) and periodic ones (daily/weekly golden set evaluation).
• Alerts based on SLOs derived from the product, not arbitrary thresholds.
• Each critical alert needs a runbook with actionable steps — an alert without a runbook is noise.
• Goodhart's Law: optimizing what you measure can break what you don't — always measure the five dimensions, not just the ones that seem important.

Checkpoint: before moving on, you should be able to:

• List the five dimensions that matter in RAG and give an example of a concrete metric for each one.
• Differentiate continuous monitoring (latency, cost) from periodic evaluation (accuracy via golden set).
• Design an alert with an SLO + runbook for p95 latency out of target.

Next capsule: 08 — Feature comparison and module summary.

You close Module 3 with a panoramic view: the features we covered (metadata filtering, hybrid search, multi-tenancy, batch ops, distance metrics, observability) and how they fit together in a production-ready RAG system. It's the summary you'll use as a quick reference when you start the hands-on in M4.

---

Resources

1. Google SRE Book — Service Level Objectives — The SLO/SLI/SLA concept
2. The Four Golden Signals (Google SRE) — Latency, traffic, errors, saturation
3. OpenTelemetry — Vendor-neutral Observability — Emerging standard for tracing
4. Honeycomb — Observability for RAG — Platform with good support for distributed tracing
5. Ragas — RAG Evaluation Framework — For automating golden set evaluation
6. Goodhart's Law — Wikipedia — The trap of optimizing what you measure

---

Estimated time: 25-30 minutes Next: 08-features-comparison-summary.md