Actively seek disconfirmation

Overview

This guide's module 4 already taught you a version of this question —when designing a test, can it truly refute the hypothesis?—. This lesson applies the same discipline to a different moment: when reading a result, not when designing it. The central question is the same in spirit, but it's asked at another instant: not "could this test fail?" but "what specific evidence, if I saw it right now, would make me change my mind?" — and the difference between asking yourself that question before seeing the result or after seeing it is, as you're going to confirm in today's example, the difference between protecting yourself from confirmation bias and being completely exposed to it.

Actively seeking disconfirmation isn't a pessimistic attitude or a way of sabotaging your own idea — it's, according to Kahneman, the opposite of what the brain does by default: intuitive thinking tends toward a "positive test strategy," seeking data compatible with what it already believes, instead of actively seeking what contradicts it. Countering that default tendency takes deliberate effort — it doesn't just happen, even with good intentions.

How this connects to the module. This lesson doesn't introduce a new formal module model (those are sampleBias() in lesson 4 and rankByStrength() in lesson 6), but it builds preRegisteredCheck(), a small teaching model that illustrates the central point: it compares the same numeric result read by two teams, one that wrote its failure condition before seeing the number, and another that didn't. It connects directly to module 4's isFalsifiable() —the same discipline of writing the wrongIf ahead of time— but now applied to the moment of reading, not designing.

An everyday analogy: the good mechanic looks for the flaw, not confirmation it "sounds fine"

You take your car to the shop because it makes a strange noise when braking. There are two very different ways a mechanic can check the problem. The first: they start the car, listen for a few seconds, say "sounds fine" and hand it back to you. The second: they take the brake apart, check the pad wear against a specific threshold, test the pedal actively looking for any sign of the flaw you described — they don't settle for "I didn't hear anything strange in the first second," but actively look for the evidence that would confirm there is a problem.

The difference isn't in the tool or the time spent — it can take similar minutes. It's in the attitude: the first mechanic is looking for a reason to say "it's fine" and hand your car back quickly; the second is actively looking for the reason why they shouldn't tell you it's fine. Reading a fake door's or an interview's result without actively seeking the evidence that would refute it is exactly the first mechanic — you can "listen" to the result, notice it "doesn't sound so bad," and hand over a conclusion that wouldn't survive closer examination.

Worked example: the same 5.2% click rate, read by two teams

recommendations's fake door (chosen in module 4, built in module 5) ends with a 5.2% click rate. It's a real number, unambiguous in itself — the ambiguity shows up in how it's read. We compare two teams: one that wrote its failure condition before running the test, and another that decides what to think after seeing the number.

// L5: preRegisteredCheck() -- compares a test's real result against the
// failure condition (wrongIf) written BEFORE running it. This lesson's
// central question --what evidence would prove me wrong-- only protects
// you if the answer gets written before seeing the result, not after.
function preRegisteredCheck(test) {
  if (!test.wrongIfWrittenBefore) {
    return { ...test, verdict: 'no protection -- no failure condition written before seeing the result' };
  }
  const failed = test.actualClickRate < test.wrongIfBelowClickRate;
  return {
    ...test,
    verdict: failed
      ? 'the hypothesis failed its own failure condition -- time to take it seriously'
      : 'the hypothesis passed its own failure condition',
  };
}

const withPreRegistration = {
  label: 'Team A -- wrote wrongIf before running the fake door',
  wrongIfWrittenBefore: true,
  wrongIfBelowClickRate: 8,
  actualClickRate: 5.2,
};

const withoutPreRegistration = {
  label: 'Team B -- decided "5.2% isn\'t so bad" after seeing the number',
  wrongIfWrittenBefore: false,
  actualClickRate: 5.2,
};

console.log('=== preRegisteredCheck() over the same result (5.2% click rate), two teams ===\n');
[withPreRegistration, withoutPreRegistration].forEach((t) => {
  const result = preRegisteredCheck(t);
  console.log(t.label + ':');
  console.log('  ' + result.verdict);
  console.log('');
});

What to expect. When you run the file with Node, the output is exactly this:

=== preRegisteredCheck() over the same result (5.2% click rate), two teams ===

Team A -- wrote wrongIf before running the fake door:
  the hypothesis failed its own failure condition -- time to take it seriously

Team B -- decided "5.2% isn't so bad" after seeing the number:
  no protection -- no failure condition written before seeing the result

The number —5.2% click rate— is identical for both teams. The only thing that changes is whether a failure condition existed, in writing, before that number existed. Team A had declared, ahead of time, that below 8% would count as evidence against the hypothesis — so when the result comes in at 5.2%, there's no room to negotiate with themselves: the hypothesis failed its own condition, written when nobody yet knew what was going to happen. Team B, with no such prior anchor, is completely exposed: it can decide, on the spot, that "5.2% isn't so bad" — a perfectly possible reading of the same number, and exactly the kind of post-hoc decision confirmation bias favors without anyone noticing.

The question, applied to a result you already have in front of you

If you already have a result on the table and never wrote a failure condition ahead of time —the more common case, honestly, on real teams—, this lesson's question still helps, though more weakly: ask yourself, right now, "if this number had come out lower than it did, at what point would I have stopped feeling comfortable with the idea?" Writing that answer before deciding what to do with the number you already have —even if it's too late for perfect pre-registration— is still better than jumping straight to "this confirms what I thought," with no intermediate question at all.

Common mistakes

Not writing down ahead of time what result would change your mind. What happens: the team runs a test —interviews, fake door, prototype— without having declared beforehand, anywhere, what number or what response pattern would count as "this refutes the idea." When the result comes in, the conversation turns into negotiating, in real time, whether the number "counts" as enough evidence. Why it happens: writing a failure condition before having any data feels like an extra, bureaucratic step, when the urgent thing seems to be running the test as soon as possible. How to spot it: ask, before running any test, "what specific result would make us abandon this idea?" — if nobody can answer with a number or a concrete criterion, there's no failure condition written down, and the later reading is unprotected, like Team B in today's example. How to fix it: adopt the same habit module 4's isFalsifiable() demands —the wrongIf— but move it to the right point on the calendar: it gets written before running the test, never after seeing the result.

Dismissing refuting evidence as "those users didn't understand." What happens: when a test's result clearly contradicts the hypothesis, someone offers an explanation that neutralizes it without subjecting it to the same scrutiny that would be demanded of favorable evidence — "those users didn't understand the prototype well," "the fake door had a bug," "there was low traffic that day." Why it happens: actively looking for why an uncomfortable result could be wrong is cognitively much easier than accepting that the hypothesis could be wrong — and there's almost always some plausible alternative explanation, which makes it tempting to use without verifying it. How to spot it: ask yourself whether you'd apply the same level of suspicion to a result that confirmed the hypothesis — would you check just as carefully whether "the prototype had a bug" when the result comes out good? If the answer is no, the scrutiny isn't even. How to fix it: any explanation that dismisses an unfavorable result must be verified with the same evidence you'd demand to accept a favorable result — it isn't enough for it to be plausible, it has to be confirmed separately, not invented to save the hypothesis.

Exercises

Exercise 1 — Run preRegisteredCheck() over a third case. A team wrote wrongIfWrittenBefore: true, wrongIfBelowClickRate: 3 before running its fake door, and got actualClickRate: 4.5. What verdict does preRegisteredCheck() return, and what does it mean for the hypothesis?

See solution

failed = 4.5 < 3 is false, so the verdict is 'the hypothesis passed its own failure condition'. Unlike the worked example, here the real result (4.5%) is above the threshold the team itself defined as failure (3%) — the hypothesis survives its own test, with a criterion written before knowing the result. This doesn't mean the hypothesis is "proven" definitively (that also depends on the evidence's strength, lesson 6's subject, and module 7's full synthesis) — it means, specifically, that this result didn't qualify as the failure signal the team had defined ahead of time.

Exercise 2 — Diagnose which team is protected. Two teams observe the same ambiguous interview result. Team X says: "we expected at least 6 of 10 users to mention the problem without us suggesting it; only 3 mentioned it, so the opportunity wasn't validated." Team Y says: "3 of 10 mentioned the problem — not bad, considering it's a new topic for them." Which team applied this lesson's discipline, and how do you know?

See solution

Team X. Their statement reveals a specific threshold ("at least 6 of 10") that was declared before knowing the real result — the same pattern preRegisteredCheck() flags as protected. Team Y, on the other hand, evaluates the number (3 of 10) with a criterion invented on the spot ("not bad, considering...") with no reference to a prior threshold — exactly the unprotected pattern of Team B in this lesson's worked example. The real number (3 of 10) is the same kind of ambiguous result in both cases; what tells the two teams apart is whether they had, ahead of time, a rule for reading it.

Exercise 3 — Apply the mechanic analogy to your own work. Describe, in two or three sentences, a situation (real or made up) where you reviewed something —code, a design, a plan— seeking to confirm it "was fine" instead of actively looking for the flaw that would knock it down. What would have changed if you had written down, before reviewing, what specific type of problem you were looking for?

See solution

There's no single answer — it depends on the example you pick —, but a good example should recognize the same structure as the mechanic: a review that, with no bad intent, stopped at "I didn't find anything strange at first glance" instead of actively looking for a specific, known flaw (an edge case, a common type of error in that context). What would change, in any case, is the same as in Mercado's example: writing down ahead of time "I'm going to specifically look for X" turns a passive review —which tends to confirm everything's fine— into an active search, with a clear goal of what would count as finding a real problem.

Summary and next step

This lesson took a question you already knew from module 4 —what result would prove me wrong?— and moved it from the moment of designing a test to the moment of reading its result. You saw preRegisteredCheck() run over the exact same number, 5.2%, read by two teams: one protected by a failure condition written ahead of time, which unambiguously recognizes the hypothesis failed; and another without that protection, completely exposed to deciding on the spot that "it isn't so bad." The number never changed — what changed was whether a prior rule existed, in writing, for reading it.

Before moving on you should be able to: explain the difference between asking yourself "what would prove me wrong?" before seeing a result and asking it afterward; and apply that question to a real ambiguous result, not just a Mercado one.

Lesson 6 gives this discipline an ally that doesn't depend on anyone's willpower: a number. You're going to build rankByStrength(), which weighs any piece of evidence by its type —observed behavior, reported behavior, stated opinion, hypothetical— no matter how convincing it felt at the moment you heard it.

Resources

• Daniel Kahneman, Thinking, Fast and Slow — us.macmillan.com/books/9780374533557/thinkingfastandslow. The source of the "positive test strategy": Kahneman describes how intuitive thinking defaults to seeking data compatible with a belief, instead of actively seeking data that contradicts it. In English.
• Paul Saffo, interview "Betting on Strong Opinions, Weakly Held" — saffo.com/interviews. The origin of the "strong opinions, weakly held" principle: form a conclusion with the available evidence, and then actively dedicate yourself to trying to knock it down — this lesson's same discipline, applied to any kind of forecast or decision. In English.
• Karl Popper, "Karl Popper" entry in the Stanford Encyclopedia of Philosophy — plato.stanford.edu/entries/popper. The same philosophical source as module 4, relevant here from another angle: Popper insisted science is distinguished by actively seeking refutation, not just by being refutable in theory. In English.