Confirmation bias

Overview

Confirmation bias is the automatic, non-deliberate tendency to seek out, notice, remember, and interpret information in a way that confirms what you already believe — and to overlook, minimize, or outright forget information that contradicts it. It isn't a character flaw or a lack of honesty: it's, according to cognitive psychology research, a structural tendency of the human brain, the same one Daniel Kahneman described as part of the normal functioning of what he calls "System 1" — fast, intuitive thinking, which forms an impression before deliberate thought has a chance to review it. Teresa Torres, who writes specifically about product discovery, cites a figure worth keeping in mind for every interview and every test result you review: your brain notices information that confirms what you already believe twice as often as information that contradicts it.

In discovery, this bias has one exact spot where it does the most damage: not at the moment of designing the test —you already protected that in module 4, with isFalsifiable() and pickTest()—, but at the moment of reading its result. A fake door with an ambiguous click rate, an interview with mixed answers, a prototype some users loved and others ignored: in all these cases, the evidence alone doesn't tell you what to conclude. And right there, exactly there, your brain —without asking your permission— tends to fill in the ambiguity in favor of what you already wanted to believe before starting.

How this connects to the module. This lesson names the entire disease the rest of the module attacks from different angles: leading questions and the say-do gap (lesson 3) are two concrete mechanisms by which confirmation bias sneaks into your data; biased samples (lesson 4) are a form of confirmation bias applied to who you interview, not just what you conclude; actively seeking disconfirmation (lesson 5) is the direct defense against this bias; and weighing evidence by its strength (lesson 6) is a way of taking away from your own judgment —the judgment confirmation bias contaminates— the power to decide alone, with no external criterion. This lesson builds recallBias(), a teaching model that simulates one of confirmation bias's most common symptoms: how memory, with no bad intent at all, retains what confirms better than what contradicts.

An everyday analogy: the horoscope that got it right

Almost nobody who reads their horoscope keeps a careful tally of how many predictions came true and how many didn't. What does happen, almost always without meaning to, is vividly remembering the day the horoscope said "today you'll have a surprise" and something unexpected really did happen — and completely forgetting the twenty days when nothing like that happened, or when the prediction was so vague that "almost anything" would have counted as a hit. Over time, that person ends up genuinely convinced "the horoscope works for me," not because they evaluated the complete evidence, but because their memory, automatically, stored the hits better than the misses.

A team that reviews recommendations's interview round "from memory," weeks later, without rereading the full notes, is in exactly the same position as the horoscope reader. They aren't lying when they say "most people reacted well" — they genuinely remember it that way. The problem is that memory, without anyone deciding it on purpose, already made the selection for the team before the synthesis conversation even started.

Worked example: recallBias() over eight interview notes

Imagine Mercado's team, after recommendations's interview round (module 2), filed away eight written notes — four confirming the team's belief ("users really do want recommendations") and four contradicting it. Weeks later, at the synthesis meeting, nobody opens the full document again: everyone "remembers" how the round went. recallBias() simulates what happens to that memory without written support.

// L2: recallBias() -- simulates what happens to a team's memory when it
// reviews its own interview notes weeks later, with no written record of
// each note at the time. Compares the real signal count against what the
// team "remembers" having heard. Teaching model: the 0.25 factor is
// illustrative, not a real measurement of human memory.
function recallBias(notes) {
  const confirming = notes.filter((n) => n.confirmsBelief);
  const disconfirming = notes.filter((n) => !n.confirmsBelief);
  const recalledDisconfirming = Math.round(disconfirming.length * 0.25);
  return {
    total: notes.length,
    confirming: confirming.length,
    disconfirming: disconfirming.length,
    recalledWithoutNotes: confirming.length + recalledDisconfirming,
    lostFromMemory: disconfirming.length - recalledDisconfirming,
  };
}

const interviewNotes = [
  { user: 'u1', confirmsBelief: true },
  { user: 'u2', confirmsBelief: false },
  { user: 'u3', confirmsBelief: true },
  { user: 'u4', confirmsBelief: false },
  { user: 'u5', confirmsBelief: true },
  { user: 'u6', confirmsBelief: false },
  { user: 'u7', confirmsBelief: false },
  { user: 'u8', confirmsBelief: true },
];

console.log('=== recallBias() over 8 "recommendations" interview notes ===\n');
interviewNotes.forEach((n) => console.log('  ' + n.user + ': ' + (n.confirmsBelief ? 'confirms the belief' : 'contradicts it')));

const result = recallBias(interviewNotes);
console.log('\nIn the written notes: ' + result.confirming + ' confirm, ' + result.disconfirming + ' contradict (out of ' + result.total + ' total).');
console.log('What the team "recalls" without rereading the notes: ' + result.recalledWithoutNotes + ' of ' + result.total + '.');
console.log('Notes that contradict the belief and got lost from memory: ' + result.lostFromMemory + '.');

What to expect. When you run the file with Node, the output is exactly this:

=== recallBias() over 8 "recommendations" interview notes ===

  u1: confirms the belief
  u2: contradicts it
  u3: confirms the belief
  u4: contradicts it
  u5: confirms the belief
  u6: contradicts it
  u7: contradicts it
  u8: confirms the belief

In the written notes: 4 confirm, 4 contradict (out of 8 total).
What the team "recalls" without rereading the notes: 5 of 8.
Notes that contradict the belief and got lost from memory: 3.

The written notes are perfectly balanced: 4 confirm, 4 contradict — a real 50/50, the kind of mixed, honest result any genuine discovery round usually produces. But what the team "remembers" without rereading them is something else: 5 of 8, with 3 of the 4 notes that contradicted the belief lost from memory. Nobody lied. Nobody discarded a note on purpose. Memory simply did its usual job — retaining more strongly what already fit the prior expectation — and the result is a version of the evidence that's no longer 50/50, but tilts, without anyone deciding it, toward confirmation.

How confirmation bias shows up in discovery

Confirmation bias doesn't appear just once, at a single moment — it seeps in at different points in the process, and it's worth naming them separately so you can watch for them:

WHERE CONFIRMATION BIAS SNEAKS IN
─────────────────────────────────────────────────────────────────
When asking the question   -> leading questions that invite a "yes"
                               (this module's lesson 3)
When listening to the      -> noticing the words that confirm more,
answer                        discounting the ones that don't fit
When taking notes          -> writing more detail for what confirms,
                               summarizing in one line what doesn't
When remembering weeks     -> recallBias(): memory retains the
later                          confirming part better, as you saw above
When reading an ambiguous  -> filling in the ambiguity in favor of
result                        what was already believed (borderline
                               fake door, mixed answers)
─────────────────────────────────────────────────────────────────

This lesson's —and the whole module's— central point is that none of these five moments requires bad faith. An entire team can act in good faith at every individual step and still end up, by the end of the process, with a version of the evidence systematically tilted toward confirmation. That's why the following lessons don't rely on "having more discipline" or "being more honest" — they rely on tools external to your own judgment: a checklist (lesson 3), a model that detects the biased sample without depending on anyone's impression (lesson 4), a question written before seeing the result (lesson 5), and a number that weighs evidence no matter how convincing it felt in the moment (lesson 6).

Common mistakes

Remembering only the interviews that confirmed. What happens: at the synthesis meeting, someone summarizes the interview round saying "most people reacted really well to the idea," based on their general memory, without reopening the full notes from each session. Why it happens: as recallBias() shows in today's example, memory without written support retains what already fit the expectation better — not out of dishonesty, but because of how human memory works with ambiguous or mixed information. How to spot it: ask "can we go back to the full notes, not the summary from memory?" — if nobody can accurately cite how many interviews confirmed and how many didn't, without rereading, the summary being used at the meeting is already biased. How to fix it: never synthesize from memory — always go back to the full written notes, and explicitly count how many confirm and how many contradict before drawing any conclusion (module 7's lesson 7 formalizes this count with synthesize()).

Dismissing refuting evidence as "those users didn't understand." What happens: when an interview or a result clearly contradicting the team's belief shows up, someone offers an explanation that neutralizes it without evaluating it — "that user isn't representative," "we didn't explain the prototype well," "they had a bad day" — and the evidence gets dismissed without the same scrutiny that would be applied to confirming evidence. Why it happens: rejecting one's own belief feels uncomfortable; finding an external reason to dismiss the data that threatens it resolves that discomfort without having to change your mind. How to spot it: ask yourself whether you'd apply the same level of scrutiny to an interview that confirmed the belief — if you never ask "could this user not be representative?" about that one, but you do ask it about the one that contradicts it, the scrutiny isn't even. How to fix it: before dismissing any evidence, apply to it the same question you'd apply to evidence you like: is there a real, verifiable reason to dismiss it, or is it just uncomfortable? If the reason wouldn't hold up equally well to dismiss confirming evidence, it isn't a real reason.

Exercises

Exercise 1 — Calculate recallBias() with a different ratio. If, instead of 8 notes (4 confirm, 4 contradict), the team had 12 notes (5 confirm, 7 contradict), and the same 0.25 recall factor for what contradicts, how many notes would the team "remember" without rereading them, and what proportion of the total would that be?

See solution

recalledDisconfirming = Math.round(7 * 0.25) = Math.round(1.75) = 2. recalledWithoutNotes = 5 + 2 = 7 out of 12 — roughly 58%, presented from memory as if "the majority confirms," when in the real notes the majority (7 of 12) actually contradicted the belief. This exercise shows a more extreme case than the worked example: even when the real evidence is mostly against, memory without written support can flip the reading toward confirmation.

Exercise 2 — Diagnose the moment of bias. A coworker tells you: "We interviewed ten users, asked open questions with no leading ones, took complete notes for each session — but at the meeting, two weeks later, we only talked from memory, without going back to the documents." Which of this lesson's five diagram moments carries the bias risk, and which don't?

See solution

The risk is specifically at "when remembering weeks later" — the diagram shows that's exactly where recallBias() acts. The other moments described ("when asking the question," with open, non-leading questions, and "when taking notes," with complete notes from each session) seem well handled, according to what the coworker describes. This case's specific mistake isn't in how the evidence was collected —that was done well— but in synthesis relying on memory without going back to the full notes, exactly this lesson's first common mistake.

Exercise 3 — Apply the horoscope analogy to your own case. Think of a belief you hold (about a product, a tool, a person, anything) that formed, at least in part, from remembering the cases that confirmed it more than the ones that didn't. Describe it in two or three sentences, and explain what evidence, if you reviewed it fully instead of from memory, might be biased in your favor.

See solution

There's no single answer — it depends on the example you pick —, but the exercise wants you to recognize, in your own experience and not just Mercado's case, the same structure: a belief that feels solid because your memory has been reinforcing it over time, not because you've ever reviewed the full, balanced evidence. Recognizing this structure in your own life is, in practice, more useful than memorizing confirmation bias's definition — it's what's going to make you stop, next time, and ask yourself "am I remembering this, or am I really reviewing it?".

Summary and next step

This lesson named the entire disease organizing the rest of the module: confirmation bias, the automatic —not dishonest— tendency to notice, remember, and interpret information that confirms what you already believe better. You saw recallBias() run over eight perfectly balanced interview notes (4 and 4), and how, with no bad intent at all, memory without written support turns them into a version tilted toward confirmation (5 of 8 "remembered," with 3 of the 4 contrary notes lost). You also saw the full map of the five moments where this bias sneaks into the discovery process, from the question to the final reading of the result.

Before moving on you should be able to: explain confirmation bias without using the word "lying" or "dishonesty"; and name, from memory, at least three of the diagram's five moments where bias can sneak in.

Lesson 3 picks back up two concepts you already know from module 2 —leading questions and the say-do gap— and looks at them now from this new angle: as two of the most common, concrete mechanisms by which confirmation bias enters your data before you even reach the analysis stage.

Resources

• Teresa Torres, "Why You Only See What You Want to See" — producttalk.org/2013/08/confirmation-bias. The source of this lesson's cited figure —the brain notices confirming information twice as often— applied directly to product discovery. In English.
• Daniel Kahneman, Thinking, Fast and Slow — us.macmillan.com/books/9780374533557/thinkingfastandslow. The underlying framework on how fast, intuitive thinking forms impressions that deliberate thinking tends, afterward, to confirm instead of question. In English.
• Nielsen Norman Group, "Confirmation Bias in UX" — nngroup.com/articles/confirmation-bias-ux. How this same bias shows up in UX research, with concrete examples of biased questions and ignored contrary evidence. In English.