The riskiest assumption test

Overview

You already know how to write a hypothesis that can truly be refuted. The next problem is different: given that there are several ways to test it —an interview, a fake door, a clickable prototype, or building the full engine— which do you pick? This lesson names the right answer: the riskiest assumption test (RAT), the cheapest test, among those that can truly refute the hypothesis, that can be run to put the bet's riskiest assumption to the test.

Notice the two words that matter in that definition, and in what order they matter. First, "among those that can truly refute it" — this rules out from the start any test that, no matter the result, would always read as a success. Second, and only among those that survived that first filter, "the cheapest" — not the most complete, not the one that feels the most rigorous, not the one that builds the final version "just to be sure." The RAT's goal isn't total certainty; it's the cheapest evidence that truly puts the belief at risk.

How this connects to the module. This lesson introduces pickTest(assumption, candidateTests), the module's second central tool, over recommendations's real list of candidate tests. You're going to reuse this function, unchanged, in lesson 5; combined with isFalsifiable() in lesson 6; and expanded with a new dimension —the strength of evidence— in lesson 7. The final version, with all its upgrades, is the one that runs in lesson 8's project over Mercado's full case.

An analogy: test the key before you move in

Before signing a new apartment's lease and moving all your furniture in, there's a cheap test almost nobody skips: trying whether the key opens the door. You don't hire a full moving crew "just to make sure the apartment is the right one" — that would be absurd, very expensive, and reversible only at a huge cost if you got it wrong. You put the key in, turn it, and in five seconds you have the answer you needed.

Now compare two different ways of "testing the key." The first: you ask the real estate agent, "does this key open the door well?" — and they, with all the good intentions in the world, tell you yes. It doesn't matter whether the key actually works or not: the question, as asked, is always going to give you back a comfortable "yes." The second: you put the key in yourself, in the real lock, and turn it. This one can actually fail — the key can get stuck, it can not fit, it can turn but the door won't open. Both ways of "testing" cost almost the same in time, but only one of the two can, in practice, tell you you're wrong. The riskiest assumption test is, always, the second option — never the first, no matter how cheap it looks.

Worked example: pickTest(), the first version

With recommendations's risky assumption on the table, Mercado's team has four candidate tests on the table, each with a cost (in personDays, the same unit from product-thinking-for-engineers), whether it can refute the hypothesis or not, and an estimated evidence strength (strength, from 1 to 10 — you're only going to use it in lesson 7, for now it just tags along with the data).

// L4: first version of pickTest(). Filters candidates that can truly
// refute the hypothesis (canFalsify), and among those, picks the
// cheapest one (cost).
function pickTest(assumption, candidateTests) {
  const falsifiable = candidateTests.filter((t) => t.canFalsify);
  if (falsifiable.length === 0) {
    return { assumption, chosen: null, discarded: candidateTests, reason: 'no candidate can refute the assumption -- a new one needs to be designed' };
  }
  const chosen = falsifiable.reduce((best, t) => (t.cost < best.cost ? t : best));
  const discarded = candidateTests.filter((t) => t.type !== chosen.type);
  return { assumption, chosen, discarded };
}

const recommendationsBet = {
  assumption: 'Users will buy more if they see personalized recommendations',
  risk: 0.6,
  impact: 5,
};

const candidateTests = [
  { type: 'ask_would_you_like_it', cost: 1, canFalsify: false, strength: 2 },
  { type: 'fake_door_checkout', cost: 3, canFalsify: true, strength: 6 },
  { type: 'clickable_prototype_interview', cost: 5, canFalsify: true, strength: 7 },
  { type: 'build_the_engine', cost: 40, canFalsify: true, strength: 9 },
];

console.log('=== pickTest() over "recommendations"\'s risky assumption ===\n');
console.log('Assumption: "' + recommendationsBet.assumption + '" (risk=' + recommendationsBet.risk + ', impact=' + recommendationsBet.impact + ')\n');
console.log('Candidates:');
candidateTests.forEach((t) => {
  console.log('  ' + t.type.padEnd(30) + ' cost=' + t.cost + '  canFalsify=' + t.canFalsify + '  strength=' + t.strength);
});

const result = pickTest(recommendationsBet.assumption, candidateTests);
console.log('\nChosen: ' + result.chosen.type + ' (cost=' + result.chosen.cost + ')');
console.log('\nDiscarded:');
result.discarded.forEach((t) => {
  const why = !t.canFalsify
    ? 'cannot refute the assumption (canFalsify=false) -- any result would read as success'
    : 'can refute it, but costs more than the chosen one (' + t.cost + ' > ' + result.chosen.cost + ')';
  console.log('  ' + t.type + ' -> ' + why);
});

What to expect. When you run the file with Node, the output is exactly this:

=== pickTest() over "recommendations"'s risky assumption ===

Assumption: "Users will buy more if they see personalized recommendations" (risk=0.6, impact=5)

Candidates:
  ask_would_you_like_it          cost=1  canFalsify=false  strength=2
  fake_door_checkout             cost=3  canFalsify=true  strength=6
  clickable_prototype_interview  cost=5  canFalsify=true  strength=7
  build_the_engine               cost=40  canFalsify=true  strength=9

Chosen: fake_door_checkout (cost=3)

Discarded:
  ask_would_you_like_it -> cannot refute the assumption (canFalsify=false) -- any result would read as success
  clickable_prototype_interview -> can refute it, but costs more than the chosen one (5 > 3)
  build_the_engine -> can refute it, but costs more than the chosen one (40 > 3)

Look at the two discards that matter most, because they're this whole lesson summarized in two lines. build_the_engine loses despite having the highest strength of the four (9) and being, in an intuitive sense, "the definitive test" — it loses because it costs 40 person-days, more than thirteen times what fake_door_checkout costs, and both can refute the same hypothesis. Paying 40 days for an answer 3 days can already give you isn't rigor, it's waste. And ask_would_you_like_it, despite being the cheapest of the four (cost: 1), doesn't even get to compete on price: it gets dropped in the first filter, because asking someone "would you like it?" is structurally incapable of going wrong — any answer, even an occasional "no, not really," can be read as "well, that particular person didn't like it" without the general belief ever being threatened.

Why the filter comes first, and minimizing cost comes after

pickTest() does two steps, in an order that isn't arbitrary: first it filters by canFalsify, and only after having that reduced list does it look for the lowest cost. If the order were reversed —sort everything by cost first, and only then check canFalsify— you'd run a real risk: the cheapest candidate in the whole original list might not be able to refute anything, and a quick glance at "the cheapest one" would take you straight to it before you got around to checking whether it works. Filtering first guarantees you'll never compare prices between a test that counts and one that doesn't — the ones that can't refute anything get taken out of the game before cost even enters the conversation. You're going to see, in lesson 5, exactly how real this risk is: a candidate even cheaper than fake_door_checkout, one that, if the filter weren't the first step, would win on price without deserving to.

Common mistakes

Choosing the convenient test instead of the one that refutes. What happens: between ask_would_you_like_it and fake_door_checkout, a team in a hurry picks asking directly — it's faster to set up, needs no change to the real product, and "we already did discovery" feels satisfying with just a day of work. Why it happens: a fake door requires touching the real checkout, even if minimally — it feels like more work, more technical risk, more coordination with the engineering team. Asking is, by comparison, almost free to organize. How to spot it: the chosen test could never, in principle, end with "the hypothesis was refuted" — it can only end with different degrees of "yeah, seems like it." How to fix it: before approving any test, run the canFalsify question first, like pickTest() does — if the answer is no, the test is completely out of the cost conversation, no matter how much cheaper or more convenient it seems.

Thinking "more expensive" is a synonym for "more reliable". What happens: someone argues building the full engine (build_the_engine) is the only way to be really sure, and that a fake door "doesn't count" because it doesn't use the real algorithm. Why it happens: effort feels like a signal of seriousness — forty days of work sound more rigorous than three. How to spot it: nobody can explain, precisely, what additional information build_the_engine would give about the specific hypothesis ("users buy more if they see recommendations") that fake_door_checkout can't already give, with hand-curated recommendations instead of algorithm-generated ones. How to fix it: remember that fake_door_checkout and build_the_engine, in today's example, share canFalsify: true — both can refute the same hypothesis about purchase behavior. The strength difference (6 versus 9) matters, and lesson 7 is going to give it its exact place — but it doesn't, by itself, justify paying more than thirteen times the cost when the cheapest candidate already clears the "can refute" bar.

Exercises

Exercise 1 — Predict without running Node. With this new list of candidates, which would pickTest() choose, and which would get discarded and why?

const newCandidates = [
  { type: 'survey_opinion', cost: 0.5, canFalsify: false, strength: 1 },
  { type: 'landing_page_signup', cost: 2, canFalsify: true, strength: 5 },
  { type: 'concierge_manual_service', cost: 8, canFalsify: true, strength: 8 },
];

See solution

Chosen: landing_page_signup (cost=2). survey_opinion gets discarded upfront by canFalsify: false (an opinion survey, just like ask_would_you_like_it, is structurally incapable of going wrong). concierge_manual_service can refute it (canFalsify: true), but costs 8 against landing_page_signup's 2 — it loses on price within the group of ones that count. landing_page_signup, with canFalsify: true and the lowest cost of the two valid candidates, wins.

Exercise 2 — Add a cheap candidate with no canFalsify. If you add { type: 'show_mockup_in_meeting', cost: 0.2, canFalsify: false, strength: 1 } —the cheapest of all, by a huge margin— to this lesson's original candidateTests set, does pickTest()'s chosen change? Explain why, without running Node.

See solution

No, the chosen one is still fake_door_checkout. show_mockup_in_meeting has cost: 0.2, far lower than any other candidate — but canFalsify: false takes it out of the falsifiable array at the function's first step, before its very low cost ever gets a chance to compete for the minimum. The .reduce() that looks for the lowest cost only runs over falsifiable, so a candidate discarded in the filter can never win, no matter how cheap it is. This is, in miniature, exactly the scenario lesson 5 is going to explore with a real example.

Exercise 3 — Explain the two steps' order in your own words. In two or three sentences, explain why pickTest() filters by canFalsify before comparing costs, and what concrete mistake would happen if someone tried doing it the other way around (compare costs first, check canFalsify after, "if there's time").

See solution

A reasonable answer: "If you compare costs first, the cheapest candidate in the full list wins that comparison before anyone checks whether it actually works — and if it turns out it can't refute the hypothesis, you already picked it by mistake, or you need an extra step to fix the decision. By filtering first, the candidates that can't refute anything are completely out of the group where cost gets compared, so there's never a chance of accidentally picking a test that doesn't work just because it was the cheapest." The concrete mistake avoided is exactly what Exercise 2 shows: an almost-free candidate, but incapable of failing, winning by default if the filter doesn't come first.

Summary and next step

In this lesson you defined the riskiest assumption test —the cheapest among the tests that can truly refute the hypothesis— and built pickTest(), which applies that definition in two steps: filter by canFalsify, and only after that minimize cost. Over recommendations's four candidates, the function chose fake_door_checkout, discarding both the cheapest option of all (ask_would_you_like_it, for being unable to refute anything) and the most "complete" one (build_the_engine, for costing thirteen times more with no need to).

Before moving on you should be able to: explain, from memory, pickTest()'s two steps and the order they happen in; and predict, for a new list of candidates, which one would get chosen without needing to run the code.

Lesson 5 takes the idea one step further: it isn't enough for a test to be able to refute the hypothesis in theory — it has to be designed, from the very first draft, with the intent of seeking the evidence that would prove you wrong, not the evidence that confirms what you already believe. You're going to see, with a real candidate, how tempting it is to design (without realizing it) a test that looks rigorous but at bottom just seeks applause.

Resources

• Marty Cagan (Silicon Valley Product Group), "The Four Big Risks" — svpg.com/four-big-risks. The four-product-risks framework (value, usability, feasibility, viability) justifying why recommendations's assumption —a value risk— is the one that needs testing first, and cheaply. In English.
• David J. Bland and Alexander Osterwalder, Testing Business Ideas summary — strategyzer.com/library/testing-business-ideas-book-summary. A catalog of dozens of experiment types, each with its relative cost and its capacity to generate evidence — the direct source behind pickTest()'s logic. In English.
• Teresa Torres, "Assumption Testing: Everything You Need to Know to Get Started" — producttalk.org/assumption-testing. On how to choose, among several ways of testing an assumption, the one that gives the most useful signal at the lowest cost — the same criterion pickTest() applies in code. In English.