Module 7: The Data and Feedback Loop

3. Observability for AI: beyond the server log

Next →

Overview

By the end of this lesson you'll understand why the observability of an AI component is different from that of a classic service, and you'll have set up a dashboard that demonstrates it. The thesis is concrete and counterintuitive: a classic server log can say your AI feature is perfect —100% successful responses, healthy latency— while half its responses are bad, because the classic log measures whether the service responded, not whether it responded well. A classic service and an AI component fail in such different ways that the observability that works for one is blind to the other. The difference, again, is the LLM's probabilistic nature: a classic endpoint that returns HTTP 200 did its job; an LLM that returns HTTP 200 with a hallucinated or irrelevant response failed, but the HTTP status has no way of knowing it.

This matters because observability is the precondition of the data loop. In lesson 2 you measured the flywheel and saw that the wheel only spins if the loop closes; but before closing anything, you have to see what's failing —you can't improve what you don't measure—. And "seeing" in an AI system means capturing things a server never had to capture: how many tokens each call consumed (because tokens are cost and latency), how much it cost, how long it took, and —the new thing, what no classic log has— a quality signal: was the response good? Without that signal, your AI feature is a black box that says "I responded" without saying "I responded well", and a flywheel built on a black box has nowhere to get the snow from.

Connection with the module: this lesson installs the first piece of the loop. Lesson 2 gave you the motivation (the flywheel compounds); here you set up the instrument that makes it possible to spin it: the observability that lets you see the quality. It's the foundation of the three lessons that follow: the feedback capture (lesson 4) is how you get the quality signal; closing the loop (lesson 5) is what you do with it; routing (lesson 7) is which lever you send it to. All of that needs, first, that you can measure the quality —which is exactly what this lesson sets up—. And a connection backward: the approval_rate you'll measure here is the live, per-request version of what module 3's eval-set measures in aggregate, against fixed cases. The eval tells you whether the component is good against your test set; the observability tells you whether it's good in production, right now. The two numbers together are the complete quality system.

Analogy: the car dashboard that only watches the engine

Imagine a car whose dashboard has only three lights: engine on, fuel, and temperature. It's a good dashboard for knowing whether the car works —the engine starts, there's fuel, it doesn't overheat—. Now get in that car for a long trip. The dashboard is all green: engine fine, fuel full, temperature normal. And yet, you're going in the wrong direction, at 20 km/h on a 120 highway, and with a tire about to blow. None of that does the dashboard see, because the dashboard was designed to watch the engine, not the trip. All green doesn't mean "you'll arrive fine"; it means "the engine is fine". They're different things.

The classic server log is that dashboard. It watches the system's "engine" —did the service respond (2xx status)?, how fast (latency)?, did it go down (error rate)?— and for a classic service that's enough, because if a classic endpoint returns 200, it did what it should. But an AI feature has a "trip" the engine dashboard doesn't see: was the response relevant?, was it correct?, did the user accept it? An AI component can have a perfect engine (200, fast, no crashes) and go in the wrong direction (bad responses). Observability for AI is adding the trip lights to the dashboard: not only "the engine runs", but "we're going well". And the most important of those lights —the one this module calls the quality signal— is the one that tells you whether the responses actually help, the only one that didn't come on the classic car's dashboard.

Worked example: the classic dashboard vs the AI dashboard

We're not going to say that the classic log hides the degradation: we're going to see it. We model a production flow of two Mercado features —the support agent and the semantic search— with their traces (one row per request: status, latency, tokens, cost, and the user's thumbs). Then we look at it with two dashboards: first the classic one (status + latency), which is what most teams have; then the observability for AI one (tokens + cost + the quality signal). The semantic search was left, without anyone noticing, on a cheaper model that degraded its quality. Let's see which of the two dashboards detects it.

# Lesson 03 (M7) — OBSERVABILITY for AI: beyond the server log. SIMULATED.
# Zero network, zero API, zero keys. Deterministic (fixed seed).
#
# A classic server log sees HTTP status and latency. For an AI feature that's
# NOT enough: you have to see tokens, cost and —what no classic log has—
# a QUALITY signal (approval_rate). The dashboard aggregates all that PER feature.
import random

random.seed(11)

# Cost model (consistent with M2): USD per 1000 tokens, latency in ms.
MODELS = {
    "cheap":  dict(usd_in=0.0008, usd_out=0.004, base_ms=90,  ms_per_tok=0.4),
    "strong": dict(usd_in=0.008,  usd_out=0.040, base_ms=300, ms_per_tok=3.0),
}

def call_cost_latency(model, tin, tout):
    m = MODELS[model]
    latency = m["base_ms"] + tout * m["ms_per_tok"]
    cost = (tin / 1000) * m["usd_in"] + (tout / 1000) * m["usd_out"]
    return latency, cost

# Each feature: its model, its token range, and its REAL approval rate.
# The semantic search was left on a cheaper model and its quality degraded:
# real approval_rate 0.62. The support agent is healthy: 0.88.
FEATURES = {
    "support_agent":  dict(model="strong", n=120, tin=(250, 400), tout=(80, 200),  approval=0.88),
    "semantic_search": dict(model="cheap",  n=200, tin=(20, 40),   tout=(40, 60),   approval=0.62),
}

# --- We generate the trace log (one row per request). All in memory. ---
traces = []
for feature, cfg in FEATURES.items():
    for _ in range(cfg["n"]):
        tin = random.randint(*cfg["tin"])
        tout = random.randint(*cfg["tout"])
        latency, cost = call_cost_latency(cfg["model"], tin, tout)
        # The HTTP status: the model call succeeded (200). Here's the trap:
        # the server responded 200 even though the response was low quality.
        http_status = 200
        # The quality signal: user's thumbs up/down (real approval_rate).
        feedback = "up" if random.random() < cfg["approval"] else "down"
        traces.append(dict(feature=feature, http_status=http_status, latency_ms=latency,
                           tokens=tin + tout, cost=cost, feedback=feedback))

def pct(vals, p):
    s = sorted(vals)
    return s[min(len(s) - 1, int(len(s) * p))]

def aggregate(feature):
    rows = [t for t in traces if t["feature"] == feature]
    lats = [t["latency_ms"] for t in rows]
    ups = sum(1 for t in rows if t["feedback"] == "up")
    return dict(
        calls=len(rows),
        ok_2xx=sum(1 for t in rows if 200 <= t["http_status"] < 300),
        avg_lat=sum(lats) / len(lats),
        p95_lat=pct(lats, 0.95),
        avg_tokens=sum(t["tokens"] for t in rows) / len(rows),
        total_cost=sum(t["cost"] for t in rows),
        approval_rate=ups / len(rows),
    )

print("=== VIEW 1: the CLASSIC server log (status + latency) ===")
print(f"{'feature':<18}{'calls':>7}{'2xx_ok':>9}{'avg_ms':>9}{'p95_ms':>9}")
for f in FEATURES:
    a = aggregate(f)
    print(f"{f:<18}{a['calls']:>7}{a['ok_2xx']/a['calls']:>8.0%}{a['avg_lat']:>9.0f}{a['p95_lat']:>9.0f}")
print("  Verdict of classic monitoring: ALL GREEN. 100% 2xx, healthy latencies.")
print()

print("=== VIEW 2: the AI OBSERVABILITY dashboard (adds the quality) ===")
print(f"{'feature':<18}{'calls':>7}{'avg_tok':>9}{'cost_usd':>10}{'approval':>10}  signal")
for f in FEATURES:
    a = aggregate(f)
    flag = "OK" if a["approval_rate"] >= 0.80 else "ALERT: low quality"
    print(f"{f:<18}{a['calls']:>7}{a['avg_tokens']:>9.0f}"
          f"{a['total_cost']:>10.4f}{a['approval_rate']:>10.0%}  {flag}")
print()
print("What the classic log did NOT see: semantic_search responds 200 in all,")
print("but its real approval_rate is 62% — a degraded feature, INVISIBLE")
print("to the server monitoring. Only the quality signal reveals it.")

What to expect. When you run it, the output is exactly this:

=== VIEW 1: the CLASSIC server log (status + latency) ===
feature             calls   2xx_ok   avg_ms   p95_ms
support_agent         120    100%      709      879
semantic_search       200    100%      110      114
  Verdict of classic monitoring: ALL GREEN. 100% 2xx, healthy latencies.

=== VIEW 2: the AI OBSERVABILITY dashboard (adds the quality) ===
feature             calls  avg_tok  cost_usd  approval  signal
support_agent         120      460    0.9649       92%  OK
semantic_search       200       80    0.0452       65%  ALERT: low quality

What the classic log did NOT see: semantic_search responds 200 in all,
but its real approval_rate is 62% — a degraded feature, INVISIBLE
to the server monitoring. Only the quality signal reveals it.

Read the two views calmly, because the comparison is the whole point of the lesson.

The classic dashboard says: everything perfect. In view 1, the two features look impeccable: 100% of 2xx responses in both, healthy latencies (the agent at 709 ms average, the search at 110 ms, both within reasonable module 2 budgets). If your monitoring were this one —and most teams' is— you'd sleep soundly: the service responds, it's fast, it doesn't crash. The verdict is "all green". And it's a verdict honest about what it measures: the engine runs well. The problem isn't that the classic dashboard lies; it's that it looks at the wrong part.

The AI dashboard says: a feature is degraded. In view 2 the column that changes everything appears: approval_rate, the quality signal. The support agent is doing well (92% approval, above the 0.80 threshold → OK). But the semantic search has an approval_rate of 65% —far below the threshold— and the dashboard marks it with ALERT: low quality. It's exactly the same search that in view 1 looked perfect (100% 2xx, 110 ms). The server responded 200 in all 200 requests; in 70 of them the response was bad. The server monitoring had no way to see it, because a bad response is also an HTTP 200. Only the quality signal —which someone had to design to capture it— reveals that this feature is failing a third of its users.

And the AI dashboard also gives you the cost and the tokens. Notice the other two new columns, which the classic log also doesn't have: avg_tok and cost_usd. The support agent consumes 460 tokens average per request and spent $0.96 on the 120 calls; the search, with a cheap model and short prompts, consumes 80 tokens and spent $0.045 on 200 calls. Those numbers aren't decorative: they're module 2's metrics (cost and latency) observed live, and they're the ones that let you detect, for example, that a prompt change doubled the tokens (and the cost) without anyone noticing. An AI component costs per token, so observability for AI has to count tokens —it's money running—. The classic log never counted tokens because a classic service doesn't charge for them.

The lesson in one sentence: the HTTP status says whether the service responded; the quality signal says whether it responded well; and for an AI feature, the two things are different. Without the second, you have a feature that can degrade for weeks with the dashboard green.

What observability for AI measures (and why each thing)

The example showed the columns; it's worth understanding why each one is necessary for an AI component and wasn't for a classic service.

The operational metrics: latency and status (what you already had). The latency and the 2xx/5xx status don't disappear —they're still necessary, a down or slow model is still a problem (module 5)—. Observability for AI includes the classic dashboard; it doesn't replace it. What it does is add the columns that are missing.

The tokens: the metric that is money and latency at once. Each LLM call consumes input and output tokens, and tokens are directly cost (you pay per token) and directly latency (more output tokens, more generation time —you saw it in module 2's cost model—). That's why observability for AI counts tokens per request and aggregates them per feature: a jump in the average tokens is a jump in the bill and in the response time. It's the metric that connects this lesson with module 2's budgets: the cost_budget and the latency_budget are watched with this observability.

The cost: the bill, broken down by feature. Aggregating the cost per feature answers the question no classic dashboard could answer: "how much does this AI feature cost me a month, and which of my features is eating the budget?". It's what lets you detect that a low-traffic but expensive-model feature costs more than a high-traffic cheap-model one, or that a prompt change triggered the cost. The cost per token, observed, is what makes module 2's cost budget governable.

The quality signal: the truly new thing. This is the column that has no equivalent in a classic server, and the reason for the lesson. The quality signal is a measure —aggregated, live— of whether the component's responses are good. In the example we modeled it as approval_rate (fraction of thumbs_up), but it can take many forms depending on the feature: the rate of relevant clicks in a search, the fraction of responses a human agent sent without editing, the escalation-to-a-human rate. Lesson 4 develops these signals in depth. What matters here is the architectural idea: without a quality signal in your observability, your AI feature is a black box that reports "I responded" without reporting "I responded well", and none of the lessons that follow —closing the loop, routing the feedback— is possible, because you have nothing to start from.

A note on the relationship with module 3's eval, because it's easy to confuse them. The eval measures the quality in aggregate, against a fixed set of cases, before the deploy —it's the gate—. The observability measures the quality live, over the real traffic, after the deploy —it's the monitoring—. They're complementary: the eval lets you not deploy something bad; the observability lets you detect that something good degraded in production (from drift, from a data change, from a model the provider updated). And the bridge between the two is the loop: the observability detects a bad case live, and that case is fed back to the eval-set (lesson 5), closing the circle.

Common mistakes

Logging like a classic server, with no quality signal (of mental model). What happens: it's the lesson's central error. The team monitors the AI feature with the same stack it uses for its classic services —status, latency, error rate— and never adds a quality signal. The feature degrades (a model change, drift, a prompt someone touched) and the dashboard stays green because bad responses also return 200. The problem is discovered from customer complaints or the drop of a business metric, weeks later. Why it happens: the instinct is to reuse the monitoring you already have, and that monitoring never had to measure "was the response right?" because classic services don't fail that way. How to detect it: if your AI feature's dashboard doesn't have a quality column, you're not observing the part that matters. How to fix it: add the quality signal (approval_rate or your feature's equivalent) to the dashboard; the status says whether it responded, the quality says whether it responded well.

Not counting tokens and discovering the bill at month's end (of omission). What happens: the team doesn't instrument the tokens per request, so it doesn't see the cost live. A prompt change that adds instructions, or a model that started giving longer responses, doubles the average tokens —and the cost— with no alarm, and the surprise comes in the provider's bill at month's end. Why it happens: a classic service doesn't charge per token, so the instrumentation instinct doesn't include counting them. How to detect it: if you can't graph the average tokens per feature over time, you won't see a cost jump coming. How to fix it: count input and output tokens per request and aggregate them per feature; it's the metric that watches module 2's cost budget live.

Confusing observability with the eval, and having only one (of scope). What happens: the team has a good eval-set (module 3) and believes that with it "we already measure the quality", so it doesn't set up observability in production; or the other way around, it has a good approval_rate dashboard and believes it doesn't need an eval before the deploy. In the first case, it doesn't detect when the feature degrades live (the eval is against fixed cases, not against the real traffic). In the second, it lets bad changes into production that an eval would have blocked. Why it happens: the two measure "quality" and sound redundant. How to detect it: if you can't say both "does this change pass the gate before the deploy?" and "what's the live approval_rate of this feature today?", you're missing one of the two. How to fix it: have both —the eval is the pre-deploy gate, the observability is the post-deploy monitoring—, and connect them with the loop (lesson 5).

Exercises

Exercise 1 — The car dashboard. Translate the analogy to design. (a) What does "the engine runs well" (engine, fuel, temperature green) represent in the observability of an AI feature? (b) What does "we're going in the right direction and at good speed" represent? (c) In the worked example, which dashboard light was green for the semantic search and which one red, and why did the classic dashboard only see the green one?

See solution
  • (a) "The engine runs well" → the operational metrics: 2xx status, latency, error rate. They're the ones that say the service works —it responded, fast, without crashing—. It's what the classic dashboard measures, and it's necessary (a down or slow model is still a problem, module 5).
  • (b) "We're going well" → the quality signal: approval_rate (or relevant clicks, or responses accepted without editing). It's the one that says whether the component does its job well, not just whether it responded. It's the light the classic dashboard didn't have.
  • (c) For the semantic search, the "engine" light was green (100% 2xx, 110 ms) and the "quality" light red (approval_rate 65%, below the threshold). The classic dashboard only saw the green one because it was designed to watch the engine (status, latency), and a bad response is also an HTTP 200 —the engine "runs" even though the response is useless—. Only a dashboard with the quality light sees that the feature is going in the wrong direction.

Exercise 2 — The invisible feature. In the example, the semantic search had 100% of 2xx and an approval_rate of 65%. A colleague says: "the 100% of 2xx proves the feature works; the approval_rate is subjective and shouldn't alarm us". Explain why both statements are wrong, and what should be done with that feature.

See solution

The first statement —"the 100% of 2xx proves it works"— confuses it responded with it responded well. The 100% of 2xx only proves the server returned a successful HTTP response in all requests; it says nothing about whether those responses were useful. A search that returns irrelevant results with status 200 is a "successful" 2xx and a failure for the user. In an AI component, the HTTP status and the response quality are independent dimensions: you can have 200 with garbage.

The second —"the approval_rate is subjective"— is also wrong. The approval_rate is an aggregated and objective measure of a real signal: what fraction of users marked the response as good (or, in the search, clicked a relevant result). It's not an opinion of an engineer looking at examples; it's the judgment of hundreds of real users, counted. That a third of users reject the responses isn't "subjective": it's a hard datum that the feature is failing a third of its traffic.

What should be done: investigate and close the loop. The quality alert (65%) is the trigger; the next step is to capture which cases fail (lesson 4), turn them into eval-set cases (lesson 5), and route them to the correct lever —here, probably, revert the change to a cheaper model that degraded the relevance, or improve the retrieval (lesson 7)—. The observability doesn't fix the feature; it detects that it has to be fixed, which is its job. Without it, the degradation would have stayed invisible.

Exercise 3 — Which quality signal for each feature? The quality signal takes different forms depending on the feature. For each of these Mercado features, propose a quality signal measurable in production (not an eval of fixed cases, but something you can count from the real traffic) and explain what it captures. (a) The support agent that proposes answers to a human agent. (b) The semantic product search. (c) The "describe your product" generator for sellers.

See solution
  • (a) The support agent → the rate of responses the human agent sent without editing (or its complement, the correction_rate). If the human sends the proposal as is, the component got it right; if they rewrite it, it failed. It's an action signal, free (it arises from the normal workflow) and honest. The final customer's thumbs and the escalation rate also work (how many tickets the agent couldn't resolve?). Lesson 4 develops these signals.
  • (b) The semantic search → the relevant_click_rate: fraction of searches where the user clicked a top-3 result (or the average click position). A click high up = the search put the relevant thing where the user sees it; a click low down or a query reformulation = the search buried the relevant thing. It's implicit feedback: the user marks nothing, their behavior (where they click) is the signal. It's the one the project uses (lesson 8).
  • (c) The "describe your product" generator → the rate of descriptions the seller published without editing, and/or how much they edited when they did. If the seller publishes the generated description as is, it helped; if they rewrite it entirely, it didn't. A more indirect signal also works (are the generated descriptions associated with more sales or views than the hand-written ones?), though that one is harder to attribute. The "accepted without editing" signal is the most direct and cheapest.

The general pattern: the best quality signal in production is usually a natural action of the user (accept, click, don't reformulate) rather than an explicit thumbs, because the action doesn't require the user to do extra work and doesn't lie. Lesson 4 goes deeper on this.

Summary and next step

In this lesson you set up the first piece of the data loop: observability for AI. You saw the thesis with the analogy of the car dashboard that only watches the engine —all green while you go in the wrong direction— and you measured it: two features with 100% of 2xx responses and healthy latencies on the classic dashboard, but the AI dashboard revealed that one of them, the semantic search, had an approval_rate of 65% —a degraded feature, invisible to the server monitoring—. You understood the columns observability for AI adds and why each one: the tokens (which are cost and latency), the cost per feature (which governs module 2's cost budget live), and —the truly new thing— the quality signal, which says whether the component responded well, not just whether it responded. And you placed the relationship with module 3's eval: the eval is the pre-deploy gate against fixed cases; the observability is the post-deploy monitoring over the real traffic.

Before moving on you should be able to: explain why a classic server log is blind to the quality of an AI component; name the columns of observability for AI (operational + tokens + cost + quality signal); distinguish the observability (live) from the eval (against fixed cases); and propose a measurable quality signal for a given feature.

What follows is the piece that produces that quality signal: the feedback loop. In lesson 4 you'll see that feedback isn't a single thing —it's three distinct signals: the explicit thumbs, the correction, and the action the human took after the suggestion— and you'll execute a feedback_loop that captures all three. You'll discover that the three don't coincide, and that the most honest one —the action— reveals hidden human work that the thumbs hides. It's the step from "I need a quality signal" to "I know exactly which signals exist, how to capture them, and why the capture point is an architecture decision".

Resources

← Previous2. The flywheel: usage becomes a better systemNext →4. The feedback loop as a design decision