AI incident postmortems: when "I couldn't reproduce it" doesn't close the case

Description

Module 6 already taught the blameless postmortem: timeline, contributing factors, action items, without blaming any person. That framework is still valid for an incident involving a model — but one step of the investigation changes in an important way, and this lesson focuses exactly on that step. Facing a code bug, any engineer's first instinct is to reproduce it: same steps, same input, does the same thing happen again? If yes, you're already 80% of the way to the root cause. Facing a model's bad output — biased, out of place, inconsistent — that same instinct can lead you to close the case as "not reproducible" when, in reality, the problem is completely real and is going to happen again.

Connection to the module. This lesson doesn't replace module 6's postmortem structure — timeline, contributing factors, action items, all of that still applies the same way. What it adds is a prior step, specific to incidents involving a model: confirming whether the responsible component is deterministic or probabilistic, before deciding whether "not reproducible" means "not real" or simply means "a different criterion is needed to confirm it."

An analogy: the photograph and the testimony

A deterministic code bug is like a photograph: you take it again under the same conditions — same angle, same light, same object — and it comes out exactly the same, no matter how many times you repeat the shot. If an engineer calls a function with the same input twice and gets different results, something is wrong with the function itself — there's a shared variable, a state that isn't being cleared, something deterministic that's broken.

A probabilistic model's output is more like a witness's testimony. You ask the same person about the same event twice, at two different moments, and it's completely normal for the second version to have different details than the first — the order in which they mention things, which detail they highlight first, some different word — even though the core of what they're telling is the same. That doesn't mean the witness is lying, or that their first testimony was false. It means you're facing a process with natural variation, and judging its reliability requires a different criterion than "tell it exactly the same way again, or I don't believe you."

An AI incident involving a probabilistic component — like a re-ranking step that uses a language model to refine a list's final order — gets investigated like the testimony, not like the photograph. The right question isn't "does it repeat exactly?", it's "how often does this pattern show up, and is it frequent enough to matter?"

Worked example: checkReproducibility() on a real re-ranking incident

For recs-v2's canary, Mercado added an extra step: a current-generation language model re-ranks the top candidates the recommendation model generates, using the buyer's recent browsing context. A user reports that, in their session, the carousel disproportionately recommended products from large retail chains, with almost no independent sellers — a pattern the team wants to investigate as a possible bias introduced by that re-ranking step. Before anything else, they confirm whether the same request, repeated, always produces the same output:

// checkReproducibility: confirms whether a reported incident reproduces the
// SAME way every time the same input repeats, or whether it varies -- the
// first question to answer before treating it like a deterministic code bug.
// Observed data from a REAL incident, captured in logs (pedagogical case).
function checkReproducibility(calls) {
  const first = calls[0].topSellerType;
  const matching = calls.filter((c) => c.topSellerType === first).length;
  const allIdentical = matching === calls.length;
  return {
    totalCalls: calls.length,
    matchingFirst: matching,
    allIdentical,
    verdict: allIdentical
      ? 'deterministic: same input, same output every time -- treat it like a code bug'
      : 'probabilistic: the same input produced different results -- "reproduce it" isn\'t enough to confirm it',
  };
}

// 5 identical calls (same user, same session context) to the re-ranking
// step, repeated to investigate the user's report.
const rerankCalls = [
  { call: 1, topSellerType: 'independent' },
  { call: 2, topSellerType: 'independent' },
  { call: 3, topSellerType: 'large-retailer' },
  { call: 4, topSellerType: 'independent' },
  { call: 5, topSellerType: 'large-retailer' },
];

console.log('=== checkReproducibility on 5 identical calls to the re-ranker ===\n');
console.log(checkReproducibility(rerankCalls));

What to expect. Running the file with Node, the output is exactly this:

=== checkReproducibility on 5 identical calls to the re-ranker ===

{
  totalCalls: 5,
  matchingFirst: 3,
  allIdentical: false,
  verdict: 'probabilistic: the same input produced different results -- "reproduce it" isn\'t enough to confirm it'
}

Five identical calls — same buyer, same context, same moment in the code that builds the request — produced three results where the top recommended seller was independent and two where it was large-retailer. No engineer who only tries to reproduce the user's report once, or even twice, is going to get a reliable answer: they could "confirm it" on the first try, or "fail to reproduce it" and close the case, depending on which of the five calls they happened to land on. This example's correct conclusion isn't "the bug exists" or "the bug doesn't exist" — it's verdict: 'probabilistic': the responsible component can give different answers to the same input, so the criterion for investigating the report has to be a different one.

How the investigation changes when the verdict is "probabilistic"

Confirming a component is probabilistic doesn't close the postmortem — it redirects it. Instead of hunting for the line of code that causes the behavior (the right approach for a deterministic bug), a probabilistic incident's investigation moves toward three different questions, which do fit within module 6's timeline → contributing factors → action items structure:

How often does the reported pattern show up, not just whether it shows up at all? In this lesson's example, matchingFirst: 3 out of 5 doesn't tell you whether the bias toward large chains happens 60% of the time overall, or whether this sample of 5 calls happened to be unrepresentative by chance. Confirming the real frequency needs a larger volume of repeated calls — the same sufficient-sample principle you already saw with shadowCompare() in lesson 4 — not five manual attempts.

Is the pattern random, or correlated with something identifiable? A probabilistic component doesn't mean "completely random, with no pattern at all." It could be that the bias toward large chains shows up more often for certain product categories, or for buyers with a certain type of browsing history — information that guides the diagnosis, exactly as shadowCompare()'s segment breakdown guided the cold-start investigation in lesson 4.

How severe is the pattern when it shows up, and what minimum frequency justifies action? A component that produces a clearly unacceptable output 0.1% of the time needs a different response than one that produces a questionable output 40% of the time. This question is, at bottom, a severity decision — the same concept module 5 already used to classify incidents — adapted to a context where "severity" and "frequency" are two independent axes that need to be evaluated together.

The final postmortem still has a timeline, contributing factors, and action items — that structure doesn't change — but the "root cause" content for a probabilistic incident is almost never a specific line of code. More often, it's a characteristic of the model's behavior under certain conditions — and the corresponding action item is rarely "fix line X"; it's more likely to be "adjust the re-ranking step to reduce variation in this type of case," "add a business rule guaranteeing a minimum seller diversity," or "lower the canary threshold until the pattern's real frequency is confirmed."

Common mistakes

Closing a ticket as "not reproducible" after a single failed attempt. What happens: an engineer gets the report, tries to reproduce it once, gets a different result than the user reported, and closes the case — without having confirmed whether the responsible component is even deterministic. Why it happens: the discipline of "reproduce it or it's not a real bug" is correct and valuable for the vast majority of software, and applying it unadjusted to a probabilistic component is a transfer error, not bad intent. How to spot it: if a ticket related to a model's output gets closed after a single reproduction attempt, with no mention of whether the involved component is deterministic or probabilistic, this mistake already happened. How to fix it: as in this lesson's example, any investigation of this kind's first question is checkReproducibility() — with several attempts, not just one — before deciding whether "not reproducible" means anything.

Confusing "it's probabilistic" with "it can't be investigated or fixed." What happens: the team confirms the responsible component varies between calls and concludes that, therefore, there's nothing concrete to do — the report gets archived as "that's just how models are." Why it happens: once it's accepted that something isn't 100% predictable, it's tempting to treat it as completely out of control, instead of as something that can be measured and bounded with the right tools. How to spot it: if the postmortem ends with no concrete action item, only the note "the model is probabilistic, there's no bug to fix," the investigation stopped too soon. How to fix it: probabilistic doesn't mean patternless — it means the pattern's frequency and correlation need to be measured (the previous section's two questions) before deciding what action to take, exactly as any postmortem would with a root cause that's harder to isolate.

Applying the same severity threshold to a probabilistic incident as to a deterministic one, without adjusting for frequency. What happens: the team classifies an incident where the reported pattern shows up in 1 out of every 500 calls with the same severity as a bug that breaks 100% of the time, because "it's a real problem either way." Why it happens: once a pattern's existence is confirmed, it's easy to treat its mere existence as sufficient for maximum urgency, without factoring in how frequent it is in practice. How to spot it: if two incidents with very different frequencies (1 in 500 versus 2 in 5) get exactly the same severity and the same response urgency, this adjustment is missing. How to fix it: a probabilistic incident's severity depends on two axes, not one: how bad the pattern is when it shows up, and how often it shows up — and both need to be measured, not assumed, before deciding the response's urgency.

Exercises

Exercise 1 — Interpret a different result. If checkReproducibility() ran on 8 calls and the result were { totalCalls: 8, matchingFirst: 8, allIdentical: true, verdict: 'deterministic...' }, what would you tell the team about how to investigate this case, compared to the lesson's example?

See solution

With allIdentical: true, the verdict is deterministic: the same input produced, all eight times, exactly the same result. In that case, the investigation should go back to the traditional code-debugging approach — finding the specific condition or line that produces that result, given that exact input — not the frequency-and-correlation approach this lesson develops for the probabilistic case. This lesson's distinction exists precisely to point the investigative effort down the right path from the start, instead of applying the same process to both types of cause.

Exercise 2 — Design a bigger sample. The team wants to confirm the real frequency of the bias-toward-large-chains pattern with more confidence than the example's 5 calls. What would you change about checkReproducibility()'s approach to use it on, say, 200 repeated calls to the same request, and what additional information would that volume give you that 5 calls can't?

See solution

The function itself doesn't need to change — it still counts how many calls match the first one — but the volume does matter: with 200 calls instead of 5, the resulting percentage (matchingFirst / totalCalls) gets much closer to the pattern's real frequency across the population of possible calls, the same way lesson 4 showed that 1,000 synthetic users give a more reliable read on a percentage than 10. With 5 calls, a result of 3 of 5 (60%) has too wide a margin of error to decide anything — with 200 calls, a similar result (say, 118 of 200, 59%) already supports a reasonable conclusion about how often the pattern shows up in practice.

Exercise 3 — Write the postmortem summary. In 3-4 sentences, and using this lesson's vocabulary (deterministic, probabilistic, frequency), write the initial summary of a postmortem for the bias-toward-large-chains incident described in the worked example, assuming a 200-call sample confirmed a 35% frequency.

See solution

A reasonable summary: "A user reported that the recommendations carousel, during recs-v2's canary, disproportionately showed products from large retail chains. We confirmed, with checkReproducibility() on 200 identical calls, that the re-ranking step is probabilistic: the reported pattern shows up in approximately 35% of calls for this type of session, neither deterministically nor completely at random. This postmortem investigates the conditions under which the pattern becomes more frequent, and defines the changes needed to the re-ranking step before continuing the canary." This summary names the problem's probabilistic nature from the start, instead of leaving it implicit, and gives a concrete frequency number instead of just confirming the pattern "exists."

Summary and next step

In this lesson you built checkReproducibility() and ran it on a real bias incident in Mercado's re-ranking step: out of 5 identical calls, only 3 matched, confirming the responsible component is probabilistic, not deterministic. You saw how that single verdict changes the rest of the investigation — from "find the line of code" to "measure the pattern's frequency and correlation" — without abandoning the blameless postmortem structure module 6 already built.

Before moving on you should be able to: explain the difference between investigating a deterministic bug and a probabilistic incident; use checkReproducibility() as the first step of any investigation involving a model; and justify why a probabilistic incident's severity depends on two axes — severity and frequency — not just one.

Lesson 6 shifts to something almost no AI incident mentions until it's already too late: what buyer data ended up saved in the logs used to investigate this very incident, and whether it all needed to be saved.

Resources

• Google SRE Book, Chapter 15, "Postmortem Culture: Learning from Failure" — sre.google/sre-book/postmortem-culture. The reference chapter on blameless postmortems module 6 already developed in depth; this lesson specifically adapts its root-cause investigation step to the case of a probabilistic component. In English.
• AI Incident Database — incidentdatabase.ai. A real repository of AI incidents documented in production, explicitly inspired by aviation safety databases — the same discipline of learning from past failures, applied to AI systems. In English.