Making it truly falsifiable

Overview

In 1934, philosopher Karl Popper proposed an idea that today is the standard for telling a scientific statement apart from one that isn't: no matter how many times you confirm a theory, you can never prove it true with absolute certainty — the contradicting case could always show up tomorrow. But a single contrary observation can knock it down completely. Popper's classic example is "all swans are white": you can see a thousand white swans and the statement is still unproven (swan number one thousand and one could be black); but you see a single black swan, and the statement is refuted, no argument possible. This asymmetry —confirming is weak and endless, refuting is strong and happens in one blow— is the underlying reason this module insists so much on the failure condition, and not on the success signal.

The previous lesson gave you a tool —isFalsifiable()— for checking whether a hypothesis technically has a wrongIf field with content. But "has content" and "can actually be refuted" aren't exactly the same thing, and in this lesson you're going to see cases where the automatic check says yes, while an attentive human reading —with Popper's criterion in mind— would say no.

How this connects to the module. This lesson reuses isFalsifiable() exactly as it stood in lesson 2, with no changes to the code at all, over three new hypotheses from recommendations's assumption stack. The goal isn't to extend the tool —that isn't needed yet— but to calibrate your own judgment for the cases where the tool, being simple, gets fooled. That refined judgment is exactly what you're going to need in lesson 4, when you have to judge whether a candidate test can truly refute the risky assumption, or only looks like it can.

An analogy: the black swan and the complaint that never arrives

Think of the statement "our users always like what we launch." Like the white swan, every successful launch "confirms" the statement — and there's never a failed launch big enough to knock it down for good, because you can always say "that one doesn't count, it had a technical problem" or "that time marketing failed, not the product." The statement is designed, without anyone doing it on purpose, to never lose. That doesn't make it safer — it makes it useless as a guide for deciding anything, exactly like "all swans are white" is useless for predicting the next swan's color if any color can be rationalized as "an exception, not a refutation."

Compare it to "sellers won't mind their products showing up less" and its failure condition, as you're going to see it in today's example: "if something goes wrong, we'll notice." It sounds like it has a failure condition — it has the word "wrong" right there — but it's exactly the same never-arriving-black-swan problem: it doesn't say what would count as "something going wrong," so anything that happens afterward can be read as "that's not what we meant by 'wrong'." It's a fake failure condition, dressed in the right words.

Worked example: isFalsifiable(), unchanged, over three new cases

We reuse the exact same code from lesson 2 — not a single different line — over three new hypotheses, taken from other assumptions in recommendations's assumption stack: whether the engine can generate relevant recommendations with current data, whether sellers won't mind the reduced visibility, and whether showing recommendations doesn't slow checkout down.

// L3: isFalsifiable() reused unchanged (identical to lesson 2), over 3 new
// hypotheses from "recommendations"'s assumption stack.
function isFalsifiable(hypothesis) {
  const hasBelief = typeof hypothesis.believe === 'string' && hypothesis.believe.trim().length > 0;
  const hasFailureCondition = typeof hypothesis.wrongIf === 'string' && hypothesis.wrongIf.trim().length > 0;
  if (!hasBelief) {
    return { ...hypothesis, falsifiable: false, reason: 'does not declare a clear belief (believe) -- there is nothing to test' };
  }
  if (!hasFailureCondition) {
    return { ...hypothesis, falsifiable: false, reason: 'does not declare a failure condition (wrongIf) -- cannot be refuted' };
  }
  return { ...hypothesis, falsifiable: true, reason: 'has both a belief and a failure condition, both observable' };
}

const engineRelevance = {
  believe: 'The engine can generate relevant recommendations with current purchase data',
  weWillKnowIf: 'at least 70% of recommendations in a sample of 30 users are relevant according to a human evaluator',
  wrongIf: 'fewer than 70% of the recommendations in the sample are relevant according to the human evaluator',
};

const vendorConcern = {
  believe: "Sellers won't mind their products showing up less in search because of the recommendations",
  weWillKnowIf: "we won't get complaints",
  wrongIf: "if something goes wrong, we'll notice",
};

const checkoutSpeed = {
  believe: 'Showing recommendations will not make checkout slower',
  weWillKnowIf: 'we measure checkout load time',
  wrongIf: '',
};

console.log('=== isFalsifiable() reused, over 3 new hypotheses from the assumption stack ===\n');
[
  { label: 'engineRelevance', h: engineRelevance },
  { label: 'vendorConcern', h: vendorConcern },
  { label: 'checkoutSpeed', h: checkoutSpeed },
].forEach(({ label, h }) => {
  const r = isFalsifiable(h);
  console.log(label + ':');
  console.log('  wrongIf: "' + (h.wrongIf || '(empty)') + '"');
  console.log('  falsifiable: ' + r.falsifiable + '  -> ' + r.reason);
  console.log('');
});

What to expect. When you run the file with Node, the output is exactly this:

=== isFalsifiable() reused, over 3 new hypotheses from the assumption stack ===

engineRelevance:
  wrongIf: "fewer than 70% of the recommendations in the sample are relevant according to the human evaluator"
  falsifiable: true  -> has both a belief and a failure condition, both observable

vendorConcern:
  wrongIf: "if something goes wrong, we'll notice"
  falsifiable: true  -> has both a belief and a failure condition, both observable

checkoutSpeed:
  wrongIf: "(empty)"
  falsifiable: false  -> does not declare a failure condition (wrongIf) -- cannot be refuted

checkoutSpeed is the easy case: wrongIf is empty, the code catches it unambiguously, done. engineRelevance is the ideal case: a concrete threshold (70%), measurable by a human evaluator over a defined sample. But look at vendorConcern carefully — isFalsifiable() marks it true, exactly like engineRelevance, because the code only checks whether wrongIf has text, not whether that text describes something observable. "If something goes wrong, we'll notice" has text — it passes the check — but it doesn't say what counts as "wrong," or how they'd notice, or when. In practice, it's just as unfalsifiable as having no wrongIf at all — it's just that the automatic heuristic has no way of knowing that, because you never taught it to read the sentence's content, only its presence.

Popper's test, applied by hand: two questions the code can't ask

When isFalsifiable() says true, it's still worth asking yourself, by hand, two questions no automatic check of this kind can answer for you:

1. Does the wrongIf describe a specific, measurable observation —a number, a threshold, a concrete event— or a vagueness that allows for many readings? engineRelevance passes: "fewer than 70%" is an exact number. vendorConcern doesn't pass: "something goes wrong" could mean anything, from a formal complaint to a hallway comment.
2. If the test's result were ambiguous, would you be tempted to say "that doesn't count, that's not what I meant"? If the answer is yes, the failure condition isn't really fixed — it can move after seeing the result, which is exactly what Popper points to as an unfalsifiable statement's trap: it isn't that you can never be wrong, it's that nobody decided ahead of time what would count as being wrong.

vendorConcern, rewritten with those two questions in mind, would look more like this: wrongIf: 'at least 15% of active sellers report a formal complaint about reduced visibility within the first 4 weeks'. Now there's a number, a deadline, and a measurement channel (formal complaint, not "we'll notice") — and the two questions above get answered with a yes and a no, respectively.

Common mistakes

Trusting that isFalsifiable() === true closes the discussion. What happens: the team runs the automatic check, sees true, and considers the hypothesis review done — exactly what would happen with vendorConcern if nobody looked past the boolean result. Why it happens: an automated tool feels objective and final, and questioning its verdict seems unnecessary if "it already passed the check." How to spot it: nobody on the team can describe, in a concrete sentence with numbers, what exact observation would trigger the wrongIf — they can only repeat the vague text exactly as written. How to fix it: use this lesson's two questions as the human step that follows after the code says true, not as a replacement for the code — they're complementary, neither substitutes the other.

Confusing "nobody has refuted it yet" with "it's proven." What happens: after several successful launches, someone says "it's proven that users like what we launch" — treating the absence of a recent failure as if it were a positive, permanent proof. Why it happens: Popper's asymmetry is counterintuitive — our instinct treats "hasn't failed" as equivalent to "works," when they're actually very different states: one is accumulated, provisional evidence, the other would be a certainty that no number of observations can ever deliver. How to spot it: the phrase "it's proven that..." shows up in a product conversation with nobody mentioning what future observation could still contradict it. How to fix it: replace "it's proven" with "so far, we haven't seen the signal that would make us change our minds" — it's longer, but it's honest about what the evidence actually supports, and leaves the door open to the next observation that could change the conclusion.

Exercises

Exercise 1 — Apply the two questions to a new hypothesis. This hypothesis would pass isFalsifiable() as true. Apply the lesson's two questions to it and decide whether it can truly be refuted: { believe: 'The new checkout design improves the experience', weWillKnowIf: 'users will perceive it as easier to use', wrongIf: "if the experience gets worse, we'll know from the feedback" }.

See solution

It passes the automatic check (wrongIf isn't empty), but it fails the two human questions. Question 1: "if the experience gets worse, we'll know from the feedback" gives no number, threshold, or specific channel — "the feedback" could be a single isolated comment, a survey, or nothing at all. Question 2: it's easy to imagine the team dismissing any isolated negative comment as "one opinion, not a real signal" — the failure condition can move after seeing the result. A better version: wrongIf: 'at least 20% of a sample of 50 users report, in a post-checkout survey, that the process felt harder than before'. Now there's a number, a defined sample, and a concrete measurement channel.

Exercise 2 — Rewrite vendorConcern so it passes both questions. Using the hint the lesson gives, write your own complete version of vendorConcern's wrongIf, with a number and a deadline, and verify by hand it would still pass isFalsifiable().

See solution

A reasonable version: wrongIf: 'at least 15% of active sellers report a formal complaint, through the support channel, about reduced visibility of their products, within the first 4 weeks of launch'. Checking it against the code: believe isn't empty, wrongIf isn't empty — isFalsifiable() would still return true, exactly like the original vague version. The difference isn't in what the code detects —both versions pass the same— but in what a human reader can do with the result: the new version has a number that can be measured with no ambiguity fifteen days after launching the test; the original doesn't.

Exercise 3 — Mercado's black swan. The statement "Mercado users always care more about a low price than anything else" looks a lot like "all swans are white": every cheap-product purchase "confirms" it, and there's never enough of those purchases to fully prove it. Write a failure condition (wrongIf) that, if it happened, would clearly and unambiguously refute this statement — like Popper's black swan.

See solution

A reasonable answer: wrongIf: 'in a sample of 50 recorded purchases, at least 30% chose a product that was NOT the cheapest among the comparable options shown'. This works as a black swan because it describes a specific, measurable observation that a neutral reader couldn't rationalize as "doesn't count" — if 30% of a clear sample chooses something other than the cheapest, the absolute statement ("they always care more about price") gets refuted, no matter how many cheap purchases were seen before. The point of the exercise: statements with words like "always" or "never" are easy to confirm endlessly and hard to falsify — unless, like here, they get a concrete, measurable failure condition attached.

Summary and next step

This lesson didn't change isFalsifiable()'s code — it reused it exactly as in lesson 2 — but it sharpened your judgment for reading its results: you saw that a wrongIf can have text and still not describe anything observable, as in vendorConcern, and you learned the two questions (is it specific and measurable? could you rationalize any result as "doesn't count"?) that no automatic check could ever ask for you. You also learned the Popper asymmetry underpinning all of this: confirming never fully proves, but a single clear refutation is enough.

Before moving on you should be able to: explain why "a thousand white swans" doesn't prove all swans are white, but a single black swan does refute it; apply this lesson's two questions to any wrongIf you see, even if isFalsifiable() already said true; and rewrite a vague failure condition into a concrete, measurable version.

Lesson 4 uses this sharpened judgment to solve the next problem: given a handful of candidate tests for recommendations's risky assumption, which do you pick? There you're going to build pickTest(), the tool that filters the candidates that can truly refute the hypothesis, and among those, picks the cheapest one.

Resources

• Karl Popper, "Karl Popper" entry in the Stanford Encyclopedia of Philosophy — plato.stanford.edu/entries/popper. The full source of the asymmetry between confirming and refuting, and of falsifiability as the boundary between what's scientific and what isn't. In English.
• David J. Bland and Alexander Osterwalder, Testing Business Ideas summary — strategyzer.com/library/testing-business-ideas-book-summary. On how to design business experiments with success and failure conditions defined ahead of time, the same discipline this lesson's criterion demands. In English.
• Teresa Torres, "Assumption Testing: Everything You Need to Know to Get Started" — producttalk.org/assumption-testing. Go back to this source with a different reading: pay attention to how it describes the difference between a vague assumption and one that can truly be put to the test. In English.