Cheap vs. strong: evidence isn't binary

Overview

pickTest(), as it stood in lessons 4 and 5, always applies the same criterion: among the candidates that can refute the hypothesis, the cheapest one. That criterion worked well in every previous example, but it hides an assumption you haven't tested yet: that a cheap test and an expensive one, if both have canFalsify: true, are equally reliable. They aren't. canFalsify tells you whether a test can, in principle, return a result that contradicts the hypothesis. It doesn't tell you how convincing that result would be if it actually happens.

This lesson adds the dimension the criterion was missing: the evidence's strength (strength). You're going to see a new candidate, so cheap it beats every other one on price, with a perfectly honest canFalsify: true — and yet, it's the wrong choice, because its evidence is so weak that a single result, in either direction, barely changes what the team should believe.

How this connects to the module. This lesson expands pickTest(), adding it a second selection mode alongside the one you already know, without breaking the previous behavior — the same function, with a new capability, following the same pattern classifyQuestion() grew by, lesson by lesson, in the previous guide about interviews. This expanded version is the one that runs, with no further changes, in lesson 8's project.

An analogy: the thermometer that almost always reads the same

Imagine two thermometers. The first is precise: it measures the real temperature, with minimal margin of error, and costs a fair amount. The second is old, uncalibrated, and almost always reads something close to 20°C no matter the room's actual temperature — sometimes it goes up a couple degrees if it's very hot, sometimes it drops a bit if it's very cold, but most readings cluster near the same central number, regardless of how hot or cold the room really is.

Technically, the uncalibrated thermometer can show you a number other than 20°C — it isn't completely broken, it doesn't always read exactly the same. But if you use it to decide whether you need to turn on the heat, a single reading from it tells you almost nothing: the noise is so large compared to the real signal that any number it shows can be explained just as well by "it's really cold" as by "the thermometer's in a bad mood today." A test with very low strength is exactly this thermometer: canFalsify: true on paper, but so unreliable in practice that a single result, whatever it is, barely moves what you should believe.

Worked example: we expand pickTest() with a byValue mode

Mercado's team now has four candidates, all with canFalsify: true — including a new one, single_user_hallway_test, which consists of showing the mockup to a single person walking by the office hallway and observing their real reaction (not asking their opinion — you already know that doesn't count, lesson 5). It's cheap, and it does observe behavior, but a single person is a tiny sample: their individual reaction says very little about how Mercado buyers in general would react.

// L7: we expand pickTest() with a byValue mode (strength/cost), without
// breaking the previous behavior (the default mode is still lowest cost).
function pickTest(assumption, candidateTests, options = {}) {
  const byValue = options.byValue || false;
  const falsifiable = candidateTests.filter((t) => t.canFalsify);
  if (falsifiable.length === 0) {
    return { assumption, chosen: null, discarded: candidateTests, reason: 'no candidate can refute the assumption -- a new one needs to be designed' };
  }
  const scored = falsifiable.map((t) => ({ ...t, value: t.strength / t.cost }));
  const chosen = byValue
    ? scored.reduce((best, t) => (t.value > best.value ? t : best))
    : scored.reduce((best, t) => (t.cost < best.cost ? t : best));
  const discarded = candidateTests.filter((t) => t.type !== chosen.type);
  return { assumption, chosen, discarded, mode: byValue ? 'best strength/cost' : 'lowest cost' };
}

const recommendationsBet = {
  assumption: 'Users will buy more if they see personalized recommendations',
};

const candidateTestsV3 = [
  { type: 'single_user_hallway_test', cost: 1, canFalsify: true, strength: 1 },
  { type: 'fake_door_checkout', cost: 3, canFalsify: true, strength: 6 },
  { type: 'clickable_prototype_interview', cost: 5, canFalsify: true, strength: 7 },
  { type: 'build_the_engine', cost: 40, canFalsify: true, strength: 9 },
];

console.log('=== pickTest() in "lowest cost" mode vs "byValue" mode (strength/cost) ===\n');
console.log('Candidates (all canFalsify=true):');
candidateTestsV3.forEach((t) => {
  console.log('  ' + t.type.padEnd(30) + ' cost=' + t.cost + '  strength=' + t.strength + '  value=' + (t.strength / t.cost).toFixed(2));
});

const byCost = pickTest(recommendationsBet.assumption, candidateTestsV3, { byValue: false });
const byValue = pickTest(recommendationsBet.assumption, candidateTestsV3, { byValue: true });

console.log('\nMode "' + byCost.mode + '": picks ' + byCost.chosen.type + ' (cost=' + byCost.chosen.cost + ', strength=' + byCost.chosen.strength + ')');
console.log('Mode "' + byValue.mode + '": picks ' + byValue.chosen.type + ' (cost=' + byValue.chosen.cost + ', value=' + byValue.chosen.value.toFixed(2) + ')');

What to expect. When you run the file with Node, the output is exactly this:

=== pickTest() in "lowest cost" mode vs "byValue" mode (strength/cost) ===

Candidates (all canFalsify=true):
  single_user_hallway_test       cost=1  strength=1  value=1.00
  fake_door_checkout             cost=3  strength=6  value=2.00
  clickable_prototype_interview  cost=5  strength=7  value=1.40
  build_the_engine               cost=40  strength=9  value=0.23

Mode "lowest cost": picks single_user_hallway_test (cost=1, strength=1)
Mode "best strength/cost": picks fake_door_checkout (cost=3, value=2.00)

There's the problem, laid bare: the "lowest cost" mode —the only one you've used so far— picks single_user_hallway_test, the example's uncalibrated thermometer. It costs less than any other candidate, and it can technically refute the hypothesis... but its strength is the lowest of the four (1, against 6, 7, and 9 for the others). If that single person in the hallway reacts well, do you really believe that confirms anything about thousands of Mercado buyers? And if they react badly, would the team be willing to kill the entire bet over one person's opinion? Almost certainly not in either case — which, thinking about it with lesson 5's criterion, is a sign the result can be rationalized in any direction, exactly the problem a well-designed test is supposed to avoid.

The byValue mode fixes this by calculating value = strength / cost for each candidate and picking the highest: single_user_hallway_test has value: 1.00, but fake_door_checkout has value: 2.00 — double the evidence per unit of cost. clickable_prototype_interview (1.40) and build_the_engine (0.23) fall even further behind: the latter, in particular, has the strongest evidence of the four in absolute terms (strength: 9), but it's so expensive that its value ends up being the worst of all.

value = strength / cost doesn't replace statistical rigor — it approximates it

This number —strength / cost— is a teaching model, like everything else in this module: a simple way to reason about the balance between "how much it costs" and "how much it tells me" without needing formal statistics yet. In a real case, deciding whether an observed difference is genuine or noise requires statistical significance and a carefully calculated sample size — exactly this guide's sibling product-metrics-and-experimentation-guide's territory. value doesn't calculate that; it just gives you a fast, honest criterion for ruling out, before running anything, the tests so weak they aren't even worth subjecting to that statistical rigor afterward.

Common mistakes

A test so weak any result "confirms" it. What happens: the team picks single_user_hallway_test because it's cheap and technically can refute the hypothesis, and runs the test — but no matter the result, nobody's really willing to act on one person's opinion, so it ends up dismissed as "inconclusive" or, worse, cited as if it confirmed the hypothesis if it happened to go well. Why it happens: lessons 4 and 5's canFalsify filter only checks whether a test can fail in principle — not whether the sample or design are strong enough for a single result to actually be informative. How to spot it: before running a test, ask yourself "if the result comes out badly, would it actually change our decision?" — if the answer is "probably not, it's too small a sample," the test is too weak even if canFalsify says yes. How to fix it: use pickTest()'s byValue mode as the default criterion, not just lowest cost — a cheap but weak test almost never wins against a slightly more expensive one with much stronger evidence.

Optimizing only for cost, ignoring the strength of the evidence. What happens: a team, proud of being "efficient," boasts about always having chosen the cheapest possible test in every discovery cycle — without realizing several of those tests, while cheap, gave signals so weak the team's confidence barely changed afterward. Why it happens: cost is easy to measure and report in a retro ("we only spent 1 day"); evidence strength is harder to quantify and, without a tool like value, easy to ignore entirely. How to spot it: review the history of tests run and ask, for each one, whether the corresponding bet's confidence truly moved after the result — if not, the test was probably cheap but weak. How to fix it: adopt value = strength / cost (or its informal equivalent) as the default choice criterion, reserving "just the cheapest" only for when every valid candidate has a reasonably similar evidence strength.

Exercises

Exercise 1 — Predict both modes without running Node. With this list of candidates, what would pickTest() choose in "lowest cost" mode and what would it choose in byValue mode?

const newCandidates = [
  { type: 'anonymous_click_tracking', cost: 2, canFalsify: true, strength: 8 },
  { type: 'single_comment_on_forum', cost: 0.3, canFalsify: true, strength: 1 },
  { type: 'ab_test_small_sample', cost: 4, canFalsify: true, strength: 7 },
];

See solution

"Lowest cost" mode: single_comment_on_forum (cost=0.3). It's the cheapest of the three, and technically meets canFalsify: true, no matter how weak its evidence is (strength: 1).

byValue mode: anonymous_click_tracking (value = 8/2 = 4.00). Comparing: single_comment_on_forum has value = 1/0.3 ≈ 3.33; ab_test_small_sample has value = 7/4 = 1.75. anonymous_click_tracking wins with the highest value of the three, combining a reasonable cost with strong evidence (strength: 8, the highest in the group).

Exercise 2 — Calculate value by hand. For a test with cost: 6 and strength: 9, calculate its value. Compare it to this lesson's worked example's fake_door_checkout (cost: 3, strength: 6, value: 2.00) — which of the two would win in byValue mode?

See solution

value = 9 / 6 = 1.50. Compared to fake_door_checkout (value: 2.00), this new test loses in byValue mode despite having higher strength in absolute terms (9 versus 6) — because it also costs twice as much (6 versus 3). The exercise confirms the lesson's central point: high strength doesn't automatically win if the cost is also high in the same proportion or more; what decides it is the relationship between the two numbers, not either one on its own.

Exercise 3 — Connect it to what comes after the module. Without going into detail —that's other lessons' and other modules' job— explain in one sentence why the "strength of evidence" idea you saw today anticipates two things you're going to find later in this guide: (a) why module 6 (bias) is going to insist that observed behavior outweighs stated opinion, and (b) why module 7 (synthesis) is going to need to move a bet's confidence differently depending on each result's strength.

See solution

A reasonable answer: "If evidence has different levels of strength —as you saw today with strength—, then (a) a user's stated opinion, being generally weaker than their observed behavior, shouldn't weigh the same when deciding whether an assumption holds up; and (b) a bet's confidence should go up or down more after a high-strength result than after a low-strength one, instead of treating every discovery result as if it were worth exactly the same." No more detail is needed than this — module 6 gives the first idea its full place, and updateConfidence() in module 7 implements the second.

Summary and next step

This lesson expanded pickTest() with a second mode, byValue, which compares strength / cost instead of just cost — without breaking the previous behavior, which stays available as the default mode. You saw that the cheapest candidate of all (single_user_hallway_test) can be the wrong choice, even meeting canFalsify: true, because its evidence is so weak that a single result barely informs anything — the same uncalibrated-thermometer problem. fake_door_checkout, with the best balance between cost and strength, remains the right choice for recommendations, now backed by a more complete criterion.

Before moving on you should be able to: explain why a cheap test with canFalsify: true can still be a bad choice; and calculate value = strength / cost for any pair of numbers.

With this, the module's design part ends. Lesson 8, the project, runs the full pipeline —isFalsifiable() over recommendations's hypothesis, and pickTest() in its byValue mode, over a realistic list of candidates that includes a behavioral interview, a fake door, a clickable prototype, and building the full engine— to reach the final decision: what, truly, is Mercado's riskiest assumption test.

Resources

• David J. Bland and Alexander Osterwalder, Testing Business Ideas summary — strategyzer.com/library/testing-business-ideas-book-summary. The original catalog classifies each experiment type both by its relative cost and by how strong the evidence it produces is — the direct source of this lesson's value idea. In English.
• Teresa Torres, "Assumption Testing: Everything You Need to Know to Get Started" — producttalk.org/assumption-testing. On why not every assumption test is equally convincing, even when they technically prove the same thing. In English.
• Marty Cagan (Silicon Valley Product Group), "The Four Big Risks" — svpg.com/four-big-risks. A reminder that correctly assessing risk —what kind, and with how good evidence— is discovery's central job, beyond just picking the cheapest test. In English.