From assumption to falsifiable hypothesis

Overview

"Users are going to buy more if they see personalized recommendations" is an assumption. It sounds reasonable, it has data behind it (risk: 0.6, impact: 5), and probably most of the team agrees with it. But notice a problem that sentence, as written, doesn't solve: if sales go up in three weeks, the team is going to say "it worked, recommendations helped." If sales don't go up, someone's going to say "it's still too early," or "the algorithm needs more data," or "it was a slow month for the whole site." The assumption, as written, can't lose. Any result can be read as confirmation, even the result that, at bottom, contradicts it.

A hypothesis is the same belief, but fitted into a form that can actually lose. The template is short and always has three parts: we believe X (the belief), we'll know it if we observe Y (the signal that would confirm X), and we'll be wrong if we observe Z (the signal that would prove X false). The third part is the one almost no team writes on its own — and it's, by far, the most important of the three.

How this connects to the module. This lesson installs the object you're going to reuse unchanged for the rest of the module: a hypothesis with three fields (believe, weWillKnowIf, wrongIf), and isFalsifiable(hypothesis), the function that checks whether those three fields are complete. Lesson 3 reuses this exact function over harder cases; lesson 6 combines it with pickTest() into a checklist; and the module's project (lesson 8) writes, with this same template, the real hypothesis Mercado's team uses to decide whether it's worth building recommendations.

An everyday analogy: touching the stove

As a kid, someone probably told you "the stove is hot, don't touch it" — an assumption, passed down from one generation to the next, that you accepted or not with no evidence of your own. There are two ways to relate to that assumption. The first: believe it forever, without ever testing it, and live with a vague version of "it's probably hot" that no real experience can confirm or contradict. The second: turn it into something testable — if I touch the stove and it doesn't burn, I'm wrong — and then, yes, touch it (carefully) to actually find out.

Notice that second sentence's exact structure, because it's this lesson's full template: we believe the stove is hot; we'll know it if touching it we feel heat or get burned; we'll be wrong if we touch it and nothing happens. The third part —the failure condition— is what turns "an idea I have about the world" into "something I can actually put to the test." Without it, touching or not touching the stove changes nothing about what you believe: any sensation can be rationalized as "well, it was a bit warm, actually."

Worked example: isFalsifiable(), the first check

With the template clear, the next step is being able to verify it in code: given an object with the three fields, does it really have a failure condition, or is it missing the piece that makes it useful? isFalsifiable() asks exactly that question, over three candidate hypotheses — the good one, first done correctly, and then two flawed versions representing the two most common mistakes when writing one.

// L2: first falsifiability check. A hypothesis needs, at minimum, a
// belief (believe) AND an explicit failure condition (wrongIf).
function isFalsifiable(hypothesis) {
  const hasBelief = typeof hypothesis.believe === 'string' && hypothesis.believe.trim().length > 0;
  const hasFailureCondition = typeof hypothesis.wrongIf === 'string' && hypothesis.wrongIf.trim().length > 0;
  if (!hasBelief) {
    return { ...hypothesis, falsifiable: false, reason: 'does not declare a clear belief (believe) -- there is nothing to test' };
  }
  if (!hasFailureCondition) {
    return { ...hypothesis, falsifiable: false, reason: 'does not declare a failure condition (wrongIf) -- cannot be refuted' };
  }
  return { ...hypothesis, falsifiable: true, reason: 'has both a belief and a failure condition, both observable' };
}

const recommendationsHypothesis = {
  believe: 'Users will buy more if they see personalized recommendations',
  weWillKnowIf: 'at least 8% of users who see a recommendation add it to their cart',
  wrongIf: 'fewer than 8% of users who see a recommendation add it to their cart, even after two weeks of exposure',
};

const noFailureCondition = {
  believe: 'Users are going to like seeing personalized recommendations',
  weWillKnowIf: "we'll notice it in the support team's reaction",
  wrongIf: '',
};

const noBelief = {
  believe: '',
  weWillKnowIf: 'we measure clicks on the new section',
  wrongIf: 'fewer than 5% of users click on the new section',
};

console.log('=== isFalsifiable() over 3 candidate hypotheses ===\n');
[recommendationsHypothesis, noFailureCondition, noBelief].forEach((h, i) => {
  const r = isFalsifiable(h);
  console.log('Hypothesis ' + (i + 1) + ':');
  console.log('  we believe: "' + (h.believe || '(empty)') + '"');
  console.log('  we will know it if: "' + (h.weWillKnowIf || '(empty)') + '"');
  console.log('  we will be wrong if: "' + (h.wrongIf || '(empty)') + '"');
  console.log('  falsifiable: ' + r.falsifiable + '  -> ' + r.reason);
  console.log('');
});

What to expect. When you run the file with Node, the output is exactly this:

=== isFalsifiable() over 3 candidate hypotheses ===

Hypothesis 1:
  we believe: "Users will buy more if they see personalized recommendations"
  we will know it if: "at least 8% of users who see a recommendation add it to their cart"
  we will be wrong if: "fewer than 8% of users who see a recommendation add it to their cart, even after two weeks of exposure"
  falsifiable: true  -> has both a belief and a failure condition, both observable

Hypothesis 2:
  we believe: "Users are going to like seeing personalized recommendations"
  we will know it if: "we'll notice it in the support team's reaction"
  we will be wrong if: "(empty)"
  falsifiable: false  -> does not declare a failure condition (wrongIf) -- cannot be refuted

Hypothesis 3:
  we believe: "(empty)"
  we will know it if: "we measure clicks on the new section"
  we will be wrong if: "fewer than 5% of users click on the new section"
  falsifiable: false  -> does not declare a clear belief (believe) -- there is nothing to test

Look closely at Hypothesis 2, because it's the most common mistake of the three: it has a clear belief, it even has a weWillKnowIf — but "we'll notice it in the support team's reaction" isn't a failure condition, it's an optimistic vagueness that only describes what success would look like. Nobody wrote down what would happen if things go wrong, so isFalsifiable() marks it, correctly, as not falsifiable: no matter what happens, there's no possible result the team has agreed beforehand counts as "we were wrong." Hypothesis 3 is the opposite mistake, less common but just as real: someone writes a complete measurement condition (weWillKnowIf and wrongIf) without ever declaring what's actually believed — a test with no real claim behind it.

Why weWillKnowIf and wrongIf are almost always the same number, seen from two sides

Look at Hypothesis 1: weWillKnowIf says "at least 8%"; wrongIf says "fewer than 8%." They aren't two different observations — they're the two sides of the same line, drawn before running the test. This isn't a coincidence or a style detail: it's the correct way to write a hypothesis. If weWillKnowIf and wrongIf point at different metrics, or at thresholds that overlap, there's a gap left in the middle where any result can be interpreted as "neither confirms nor refutes, but we're probably fine" — the same trap as the original assumption, just now hidden in the fine print. Always write the threshold once, and describe the two sides of that same line. The specific number —8%, in this case— is a teaching threshold for the example; in a real case, that number comes out of the business conversation (how much does conversion need to go up for it to be worth building the full engine?), not from a statistics table.

Common mistakes

A "hypothesis" with no failure condition — it can't be refuted. What happens: the team writes "we believe users are going to like seeing recommendations" and presents it as the hypothesis ready to test, with no wrongIf at all. Why it happens: the belief on its own already feels complete — it has a subject, it has a predicate, it sounds like a serious statement — and what's missing is the less intuitive part: declaring ahead of time what would count as failure. How to spot it: exactly what isFalsifiable() did with today's Hypothesis 2 — if the wrongIf field is empty or missing, there's no hypothesis yet, just an expectation. How to fix it: don't close out writing a hypothesis until you've written its explicit failure condition, in the same session where you write the belief — never after seeing the test's results.

Writing a vague wrongIf that couldn't actually be observed in practice. What happens: someone does fill in the wrongIf field, but with something like "if it doesn't work, we'll notice" — technically there's text there, but it doesn't describe any concrete, measurable observation. Why it happens: it feels like the rule got followed ("I have my three fields filled in") without having done the real work of thinking through what specific observation would prove the error. How to spot it: ask yourself whether two different people, looking at the same test result, could reach opposite conclusions about whether wrongIf happened or not — if the answer is yes, the text is too vague. How to fix it: this module's lesson 3 digs into this specific mistake —the blind spot of an automatic check like isFalsifiable()— and gives you the criterion to spot it by eye, not just with code.

Exercises

Exercise 1 — Classify without running Node. For each of these three hypotheses, say what isFalsifiable() would return (true or false) and why:

• (a) { believe: 'The recommendations engine will not make the site slower', weWillKnowIf: 'load time stays under 2 seconds', wrongIf: 'load time goes above 2 seconds' }
• (b) { believe: 'Users are going to love the new section', weWillKnowIf: 'we get positive feedback', wrongIf: '' }
• (c) { believe: '', weWillKnowIf: '', wrongIf: 'fewer than 5% click' }

Exercise 2 — Write your own falsifiable hypothesis. recommendations's assumption stack has another assumption besides the main one: "sellers won't mind their products showing up less in search because of the recommendations." Write a complete hypothesis (believe, weWillKnowIf, wrongIf) for that assumption, and verify by hand that it would pass isFalsifiable() as true.

{
  believe: "Sellers won't mind their products showing up less in search because of the recommendations",
  weWillKnowIf: 'fewer than 10% of sellers report a visibility-related complaint in the first 4 weeks',
  wrongIf: '10% or more of sellers report a visibility-related complaint in the first 4 weeks',
}

Exercise 3 — Explain why the order of the fields matters in the code. Without running Node, what would happen if isFalsifiable() checked hasFailureCondition before hasBelief? For the worked example's noBelief case, would the final result change, or just the reported reason?

Summary and next step

In this lesson you turned an assumption —a sentence no result can contradict— into a hypothesis with structure: we believe X, we'll know it if we observe Y, we'll be wrong if we observe Z. You wrote (and ran) isFalsifiable(), the module's first tool, and saw its two ways of failing: a hypothesis with no belief, and —far more common— a hypothesis with no failure condition.

Before moving on you should be able to: write the three-part template from memory; identify, in someone else's hypothesis, whether it's missing the wrongIf field; and explain why weWillKnowIf and wrongIf almost always describe the same threshold, seen from two sides.

Lesson 3 tests isFalsifiable()'s limits: you're going to see cases where the automatic check says true — because the wrongIf field technically isn't empty — but a human reader, with Karl Popper's criterion for what makes a statement scientific, would recognize it actually can't be refuted.

Resources

• Jeff Gothelf, "The Lean UX Canvas" — jeffgothelf.com/blog/the-lean-ux-canvas. The origin of the hypothesis template this lesson uses: turning business assumptions into concrete hypotheses before designing the experiment that puts them to the test. In English.
• Teresa Torres, "Assumption Testing: Everything You Need to Know to Get Started" — producttalk.org/assumption-testing. On what, exactly, an assumption is within discovery's vocabulary, and why turning it into something testable is the first step before designing any test. In English.
• Karl Popper, "Karl Popper" entry in the Stanford Encyclopedia of Philosophy — plato.stanford.edu/entries/popper. The full source of the falsifiability criterion underpinning this lesson and the next: a statement is testable only if there's a possible observation that would contradict it. In English.