Step 1: turn the assumption into a falsifiable hypothesis

Overview

Everything that follows in this module depends on this first step, so it's worth taking it slowly, even though you already saw it in module 4. recommendations's risky assumption —"users are going to buy more if they see personalized recommendations"— arrived from product-thinking-for-engineers-guide as a loose sentence, with a risk: 0.6 and an impact: 5 marking it as the backlog's most urgent one, but with no way yet to know whether it's true. Before designing any test, any interview, any fake door, this lesson turns it into a falsifiable hypothesis: a statement with an explicit failure condition, written before collecting any data.

How this connects to the module. This lesson reuses isFalsifiable(hypothesis) exactly as it stood in module 4, with no changes at all. There's no new model here — what this step does is fix, in writing and verified, the starting point the following six steps rest on: if this lesson's hypothesis were poorly formed, pickTest() (lesson 3) would choose a test to refute the wrong statement, the interview script (lesson 4) would ask about the wrong thing, and the final decision (lesson 7) would have no solid foundation underneath it. This entire module's pipeline rests on this first step being done well.

An analogy: the bet you can lose, not the one you can't

A serious gambler never bets "I think the best team is going to win" — that sentence can't lose, because any result can be reinterpreted as "yes, the best team won." They bet something concrete: "team A wins by more than 1.5 goals," a statement that, if the result comes out different, loses unambiguously. The difference between the two sentences isn't about enthusiasm or conviction — it's about whether there's, ahead of time, an observable result that would prove the bet wrong. This lesson's hypothesis does exactly that with recommendations: not "we believe users are going to like it" (unlosable), but a concrete figure that, if not reached, leaves the assumption officially refuted.

Worked example: isFalsifiable() over two versions of the same belief

We start with the version that, in a rushed meeting, is easy to write without noticing the problem: a reasonable belief, but with no failure condition at all. We reuse isFalsifiable() exactly as it stood in module 4:

// L2 (M8): isFalsifiable() over two versions of the same belief -- the
// vague version that almost gets written by default, and the falsifiable
// version this step leaves ready for the rest of the pipeline. EXACT
// function from module 4.
function isFalsifiable(hypothesis) {
  const hasBelief = typeof hypothesis.believe === 'string' && hypothesis.believe.trim().length > 0;
  const hasFailureCondition = typeof hypothesis.wrongIf === 'string' && hypothesis.wrongIf.trim().length > 0;
  if (!hasBelief) {
    return { ...hypothesis, falsifiable: false, reason: 'does not declare a clear belief (believe) -- there is nothing to test' };
  }
  if (!hasFailureCondition) {
    return { ...hypothesis, falsifiable: false, reason: 'does not declare a failure condition (wrongIf) -- cannot be refuted' };
  }
  return { ...hypothesis, falsifiable: true, reason: 'has both a belief and a failure condition, both observable' };
}

console.log('=== The vague version, the one almost written by default ===\n');
const vagueVersion = {
  believe: 'Users will buy more if they see personalized recommendations',
  weWillKnowIf: "we'll notice it in purchase behavior",
};
const vagueCheck = isFalsifiable(vagueVersion);
console.log('falsifiable: ' + vagueCheck.falsifiable + ' -> ' + vagueCheck.reason);

console.log('\n=== The falsifiable version, the one the rest of the module uses ===\n');
const recommendationsHypothesis = {
  believe: 'Users will buy more if they see personalized recommendations',
  weWillKnowIf: 'at least 8% of users who see a recommendation add it to their cart',
  wrongIf: 'fewer than 8% of users who see a recommendation add it to their cart, even after two weeks of exposure',
};
const falsifiabilityCheck = isFalsifiable(recommendationsHypothesis);
console.log('we believe: "' + recommendationsHypothesis.believe + '"');
console.log('we will know it if: "' + recommendationsHypothesis.weWillKnowIf + '"');
console.log('we will be wrong if: "' + recommendationsHypothesis.wrongIf + '"');
console.log('falsifiable: ' + falsifiabilityCheck.falsifiable + ' -> ' + falsifiabilityCheck.reason);

What to expect. When you run the file with Node, the output is exactly this:

=== The vague version, the one almost written by default ===

falsifiable: false -> does not declare a failure condition (wrongIf) -- cannot be refuted

=== The falsifiable version, the one the rest of the module uses ===

we believe: "Users will buy more if they see personalized recommendations"
we will know it if: "at least 8% of users who see a recommendation add it to their cart"
we will be wrong if: "fewer than 8% of users who see a recommendation add it to their cart, even after two weeks of exposure"
falsifiable: true -> has both a belief and a failure condition, both observable

Look carefully at the difference between the two versions: both share exactly the same believe — the underlying belief didn't change at all. The only thing separating the vague version from the falsifiable one is a single line, wrongIf, and that line does all the work. "We'll notice it in purchase behavior" sounds reasonable in a meeting, but it doesn't say how much behavior counts as "noticing it" — any result, high or low, can be reinterpreted afterward as if it had confirmed the belief. "Fewer than 8%... even after two weeks" doesn't leave that room: it's a concrete figure, with a concrete deadline, that a real result can violate unambiguously.

Notice too the deadline detail — "even after two weeks of exposure" — that the vague version doesn't have. Without that deadline, someone could defend the hypothesis indefinitely by saying "we haven't given it enough time yet," with there never being a moment the hypothesis could be considered refuted. The deadline closes that escape hatch: two weeks is the time the team agreed on, ahead of time, as reasonable for behavior to stabilize, and after that deadline, the number rules, no new excuses.

Why the threshold gets fixed before seeing any data

weWillKnowIf's 8% isn't a number that comes out of a formula — it's a team decision, made before running any test, about how much signal it takes to justify continuing to invest in recommendations. It could have been 5% or 12%; what matters isn't the exact number, but that it got fixed before any data existed that could influence the choice. This is the same discipline you saw with module 5's fake door threshold (5% click-through, agreed on before the measurement week) — and it's no coincidence it repeats here: it's the simplest, most effective defense against the confirmation bias module 6 named in detail. A team that fixes the threshold after seeing the result can, without realizing it, move the bar exactly to wherever the number it already has lets it pass.

                    BEFORE seeing data          AFTER seeing data
                    ───────────────────         ──────────────────────
Threshold fixed     "8%, decided today"         "whatever number comes
                                                  out is the threshold,
                                                  adjusted to taste"
Result: 6%          Hypothesis REFUTED           "6% is already pretty
                    (below the threshold)         good, let's call it
                                                   a success"
Who decides         The team, with a clear       The team, already with
                    mind                          the result in front of
                                                   them, tempted to read
                                                   it however they want

Common mistakes

Writing weWillKnowIf without writing wrongIf. What happens: someone fills in the template with a success condition ("we'll know it if 8% adds it to their cart") but never explicitly writes the failure condition, assuming it's the same sentence "reversed" and that it doesn't need to be written separately. Why it happens: it seems redundant to write, twice, in two directions, what feels like the same idea. How to spot it: ask whoever wrote the hypothesis to recite, from memory, the exact wrongIf — if they hesitate, or improvise it on the spot, the hypothesis never really had a failure condition, only a success one everyone was comfortable with. How to fix it: write the two sentences separately, each with its own threshold and its own deadline — as you saw in today's example, wrongIf added the "two weeks" deadline weWillKnowIf didn't have, and that detail only shows up when someone forces themselves to write the failure condition explicitly, not as an automatic mirror of the success one.

Choosing a comfortable threshold instead of an honest one. What happens: when fixing the 8% (or any number), the team, without saying it out loud, picks one it knows is going to be easy to clear, instead of one that truly reflects how much signal the business needs to justify building the full engine. Why it happens: an easy-to-clear threshold feels like a safer bet — nobody wants to be the one who set the bar so high the project "failed" the evaluation. How to spot it: ask what would happen if the real number lands just below the chosen threshold — if the instinctive answer is "we'd move forward anyway," the threshold wasn't set honestly. How to fix it: the threshold has to reflect the real cost of building the full version (build_the_engine, 40 person-days according to module 4) versus the value it brings — not whoever writes it's comfort level. If you're unsure what number to use, go back to product-thinking-for-engineers-guide's opportunitySize to anchor the threshold in the real business, not in the feeling of "something seems reasonable."

Exercises

Exercise 1 — Find the missing wrongIf. A teammate writes this hypothesis for a different bet, sellerTools: { believe: 'Better tools make sellers upload more products', weWillKnowIf: "the pilot sellers' catalog grows faster than the control group's" }. Without running Node, what would isFalsifiable() return over this object, and why?

See solution

It would return { falsifiable: false, reason: 'does not declare a failure condition (wrongIf) -- cannot be refuted' }. The object has a believe with content (it passes the first check), but has no wrongIf property at all — hypothesis.wrongIf would be undefined, and typeof undefined === 'string' is false, so hasFailureCondition stays false and the function stops at the second if. This exercise confirms something important about isFalsifiable(): it doesn't evaluate whether the success condition is well written — it only verifies a failure condition exists, explicit and separate. A hypothesis can have the best weWillKnowIf in the world and still be falsifiable: false if it never bothered to write the opposite.

Exercise 2 — Adjust the threshold and reason about the impact. If the team decided 12% (instead of 8%) is the right threshold for recommendations —reasoning that the full engine costs 40 person-days and needs a stronger signal to justify that investment—, rewrite weWillKnowIf and wrongIf with that new threshold. Does this change whether the hypothesis is falsifiable according to isFalsifiable()?

See solution

A reasonable rewrite: weWillKnowIf: 'at least 12% of users who see a recommendation add it to their cart', wrongIf: 'fewer than 12% of users who see a recommendation add it to their cart, even after two weeks of exposure'. No, it doesn't change whether it's falsifiable — isFalsifiable() only verifies believe and wrongIf exist and have content, regardless of what the threshold's exact number is. This is intentional: the function checks the hypothesis's shape (does it have a concrete failure condition?), not whether the specific threshold is the right one for the business — that second question, about which number is correct, is a business judgment call the team has to make separately, with data like the sibling guide's opportunitySize.

Exercise 3 — Write the hypothesis for a bet you never saw in this guide. Imagine a new bet for Mercado: "adding a 'buy again' button to the order history increases repurchase frequency." Write the complete hypothesis (believe, weWillKnowIf, wrongIf), with a concrete threshold and deadline, and manually verify it would pass isFalsifiable().

See solution

A reasonable hypothesis: { believe: 'Adding a buy-again button to the order history increases repurchase frequency', weWillKnowIf: 'at least 10% of buyers who use the button make a second purchase within 30 days, compared to the control group', wrongIf: 'less than a 10% difference from the control group, even after 30 days of exposure to the button' }. It passes isFalsifiable(): it has believe with content and wrongIf with content, both observable (a measurable repurchase rate, with a concrete 30-day deadline). Notice a detail this exercise makes clear: the hypothesis compares against a control group, not against an absolute number — a more rigorous way of writing wrongIf, which rules out repurchase having gone up anyway for reasons unrelated to the button (seasonality, a simultaneous marketing campaign). It isn't required to pass isFalsifiable(), but it's a stronger practice when context allows it.

Summary and next step

In this lesson you turned recommendations's risky assumption into a falsifiable hypothesis, with isFalsifiable() confirming it has both a clear belief and a concrete, deadlined failure condition: fewer than 8% adding to cart after two weeks refutes it. You saw, comparing the vague version against the falsifiable one, that the difference between them isn't in the enthusiasm or the general wording — it's in a single line, written before seeing any data, that decides ahead of time what result counts as failure.

Before moving on you should be able to: write, for any new assumption, a hypothesis with separate, concrete believe and wrongIf; and explain why fixing the threshold before seeing data is an active defense against confirmation bias.

Lesson 3 takes this now-verified hypothesis and chooses, with pickTest(), the cheapest test that can truly refute it — among several legitimate candidates, not between the most convenient one and the rest.

Resources

• Karl Popper, "Karl Popper" entry in the Stanford Encyclopedia of Philosophy — plato.stanford.edu/entries/popper. The philosophical source behind isFalsifiable() — for anyone who wants the full argument behind "a good hypothesis is one that can be knocked down." In English.
• Teresa Torres, "Assumption Testing: Everything You Need to Know to Get Started" — producttalk.org/assumption-testing. The reference article for continuing to practice writing falsifiable hypotheses, beyond recommendations's case. In English.
• David J. Bland and Alexander Osterwalder, Testing Business Ideas summary — strategyzer.com/library/testing-business-ideas-book-summary. On why fixing the success threshold before running the test is one of the most cited —and most skipped— practices in business experiment design. In English.