Step 2: choose the cheapest test that can truly refute it

Overview

With lesson 2's falsifiable hypothesis in hand, this step answers the question module 4 formalized with a complete algorithm: of all the possible tests, which one do you run first? Not just any test that "sounds good" in a meeting — the one that best balances how much it costs against how much it can refute the hypothesis. This lesson picks back up, with no changes at all, module 4's exact decision: four legitimate candidate tests —a behavioral interview, a checkout fake door, a clickable prototype, and building the full engine— and chooses just one, with an explicit criterion, not with whichever one is most comfortable for whoever proposed it.

How this connects to the module. This lesson reuses pickTest(assumption, candidateTests, options) exactly as it stood in module 4, in its byValue mode. This lesson's choice —which test to run— directly determines the rest of the pipeline: lesson 4's interview script gets designed knowing the interview is a complement, not the main test; lesson 5's signal (fakeDoorSignal()) exists because this step, and no other, chose fake_door_checkout; and the evidence weighed in lesson 6 deliberately includes both the fake door chosen here and the discarded interview, to show that "discarded" doesn't mean "worthless."

An analogy: the key you try before moving in

Before committing to a full move, you don't hire an exhaustive inspection of the whole city, nor do you trust a neighbor's promise that "you're going to love the house" — you try the key in the lock. It's cheap, it takes a minute, and it tells you something concrete and unambiguous: the key turns or it doesn't. Choosing a discovery test works the same way: you don't pick the most exhaustive one possible (too expensive for an assumption that might still be wrong), nor the easiest one to get (an opinion that commits nobody), but the cheapest one that would truly tell you whether the key turns.

Worked example: pickTest() over recommendations's four candidates

We reuse, with no changes at all, module 4's pickTest(), in its byValue mode — the one that balances cost and strength of evidence, instead of blindly choosing the cheapest:

// L3 (M8): pickTest() in byValue mode over the 4 legitimate candidates to
// refute the "recommendations" hypothesis. EXACT function from module 4.
function pickTest(assumption, candidateTests, options = {}) {
  const byValue = options.byValue || false;
  const falsifiable = candidateTests.filter((t) => t.canFalsify);
  if (falsifiable.length === 0) {
    return { assumption, chosen: null, discarded: candidateTests, reason: 'no candidate can refute the assumption -- a new one needs to be designed' };
  }
  const scored = falsifiable.map((t) => ({ ...t, value: t.strength / t.cost }));
  const chosen = byValue
    ? scored.reduce((best, t) => (t.value > best.value ? t : best))
    : scored.reduce((best, t) => (t.cost < best.cost ? t : best));
  const discarded = candidateTests.filter((t) => t.type !== chosen.type);
  return { assumption, chosen, discarded, mode: byValue ? 'best strength/cost' : 'lowest cost' };
}

console.log('=== The 4 candidates, with their value (strength/cost) ===\n');
const believe = 'Users will buy more if they see personalized recommendations';
const candidateTests = [
  { type: 'behavioral_interview', cost: 2, canFalsify: true, strength: 3 },
  { type: 'fake_door_checkout', cost: 3, canFalsify: true, strength: 6 },
  { type: 'clickable_prototype', cost: 5, canFalsify: true, strength: 7 },
  { type: 'build_the_engine', cost: 40, canFalsify: true, strength: 9 },
];
candidateTests.forEach((t) => {
  console.log('  ' + t.type.padEnd(22) + 'cost=' + t.cost + '  strength=' + t.strength + '  value=' + (t.strength / t.cost).toFixed(2));
});

console.log('\n=== pickTest() in byValue mode ===\n');
const decision = pickTest(believe, candidateTests, { byValue: true });
console.log('Chosen: ' + decision.chosen.type + ' (cost=' + decision.chosen.cost + ', strength=' + decision.chosen.strength + ', value=' + decision.chosen.value.toFixed(2) + ')');
console.log('\nDiscarded, with their value:');
decision.discarded.forEach((t) => console.log('  ' + t.type + ' -> value=' + (t.strength / t.cost).toFixed(2)));

What to expect. When you run the file with Node, the output is exactly this:

=== The 4 candidates, with their value (strength/cost) ===

  behavioral_interview  cost=2  strength=3  value=1.50
  fake_door_checkout    cost=3  strength=6  value=2.00
  clickable_prototype   cost=5  strength=7  value=1.40
  build_the_engine      cost=40  strength=9  value=0.23

=== pickTest() in byValue mode ===

Chosen: fake_door_checkout (cost=3, strength=6, value=2.00)

Discarded, with their value:
  behavioral_interview -> value=1.50
  clickable_prototype -> value=1.40
  build_the_engine -> value=0.23

fake_door_checkout wins with the highest value of the four (2.00) — neither the cheapest in absolute terms (behavioral_interview costs 2 against 3), nor the one with the highest strength (build_the_engine reaches 9), but the one that best balances the two things. This is the exact same result you already saw in module 4's project — and the repetition is intentional: this step isn't looking for a new answer, it confirms the decision still holds now that the hypothesis became formally falsifiable in the previous lesson.

Why behavioral_interview, the cheapest, doesn't win

It's worth pausing on a number that's surprising at first glance: behavioral_interview costs less (cost: 2) than fake_door_checkout (cost: 3), and yet it loses. The reason is in strength: an interview about past behavior, even one well designed with module 2's script, is still a verbal account of past experiences with recommendations in other contexts —other stores, other apps—, not behavior observed directly in front of Mercado's real product. fake_door_checkout, on the other hand, measures a real action —a click— in the exact place where it matters. Neither test is poorly designed; they simply produce evidence of a different strength, and that gets reflected in the final value.

This doesn't discard the interview from the full pipeline — it only discards it as the main test that refutes or confirms the hypothesis. Lesson 4 picks it back up with a different role: not as a substitute for measuring real behavior, but as preparation that helps better interpret the signal the fake door is going to give.

                        cost     strength     value = strength/cost
                        ────     ────────     ──────────────────────
behavioral_interview     2          3          1.50
fake_door_checkout       3          6          2.00  <- chosen
clickable_prototype      5          7          1.40
build_the_engine        40          9          0.23

Common mistakes

Choosing the highest-strength test without looking at cost. What happens: someone looks at the candidates table and reasons "we want the strongest possible evidence, so let's build the full engine directly" — ignoring that build_the_engine costs 40 person-days for a mere 9 of strength, the worst ratio in the group. Why it happens: "stronger" intuitively sounds like "better decision" — without putting that number next to the cost of getting it. How to spot it: if a test's justification mentions only its strength, with no value calculation at all, check whether there's a cheaper candidate with a higher value. How to fix it: pickTest() in byValue mode exists exactly for this — it always compares the full value, not just one of the two columns, before committing budget.

Confusing "we chose the test" with "we already know the result". What happens: after pickTest() returns fake_door_checkout, someone on the team communicates that "recommendations is on the right track," as if choosing the right test were already evidence the hypothesis is true. Why it happens: the Node pipeline produces a decision with concrete numbers, which feels like a conclusion about the business, when it's actually a conclusion about what to measure, not about what got measured. How to spot it: ask what the fake door's real click-through rate was — if the answer is "we haven't run it yet," validation hasn't started, it's only designed. How to fix it: pickTest() answers "what do we run?"; this module's lesson 5 answers "what happened when we ran it?" — they're two different questions, and neither replaces the other.

Exercises

Exercise 1 — Add a fifth candidate and recalculate. A team member proposes { type: 'landing_page_waitlist', cost: 1.5, canFalsify: true, strength: 4 } — a page where users sign up to "be the first to try the recommendations." Calculate its value by hand and determine whether it would displace fake_door_checkout as the chosen one in byValue mode.

Exercise 2 — Compare pickTest()'s two modes. Without running Node, if you ran pickTest(believe, candidateTests) without { byValue: true } (the default mode, which chooses by lowest cost), which test would get chosen instead of fake_door_checkout? Why does the guide prefer byValue mode for this decision?

Exercise 3 — Apply the criterion to a new assumption. For the "I don't trust new sellers without reviews" opportunity (the second one in module 3's tree), propose three candidate tests with their estimated cost, canFalsify, and strength, and use pickTest() in byValue mode to choose one. There's no single correct answer — the goal is to apply the same reasoning to a new case.

const trustTests = [
  { type: 'survey_would_you_trust', cost: 1, canFalsify: false, strength: 1 }, // opinion question, refutes nothing real
  { type: 'clickable_badge_prototype', cost: 3, canFalsify: true, strength: 5 },
  { type: 'ab_test_real_badge', cost: 8, canFalsify: true, strength: 8 },
];

Summary and next step

In this lesson you ran pickTest() in byValue mode over the four legitimate candidates to refute recommendations's hypothesis, and confirmed the same result from module 4: fake_door_checkout wins with the best balance between cost and strength of evidence (value: 2.00), not for being the cheapest or the strongest in absolute terms. You also saw why behavioral_interview, despite being cheaper, loses — its evidence is a verbal account, not behavior observed in front of the real product.

Before moving on you should be able to: calculate any candidate test's value by hand; and explain, unambiguously, why choosing by pure strength or by pure cost produces worse decisions than choosing by the balance of the two.

Lesson 4 doesn't abandon the interview discarded in this lesson — it picks it back up with its correct role: preparation that helps better interpret the signal the fake door chosen here is going to give.

Resources

• David J. Bland and Alexander Osterwalder, Testing Business Ideas summary — strategyzer.com/library/testing-business-ideas-book-summary. The complete catalog of experiment types, to keep practicing the choice between candidates beyond this lesson's four. In English.
• Marty Cagan (Silicon Valley Product Group), "The Four Big Risks" — svpg.com/four-big-risks. On why a value risk like recommendations deserves this level of care before committing weeks of engineering. In English.
• Teresa Torres, "Assumption Testing: Everything You Need to Know to Get Started" — producttalk.org/assumption-testing. Revisit it here, with module 8's full pipeline already underway. In English.