Module 1: Why Discovery

What discovery buys: cheap evidence

Next →

Overview

Lesson 4 left an uncomfortable number: building recommendations without validating risks, on average, 42 person-days lost. This lesson answers the obvious question that number leaves open: what's the alternative, and how much does it cost? This is where discovery stops being an abstract idea ("talking to users is good") and becomes a concrete comparison, with the same numbers you used in the previous lesson.

Discovery doesn't buy absolute certainty —only measuring in production for months gives you that, at best—. What it buys is evidence: a real signal, about real behavior, strong enough to decide whether the full version is worth building. And it buys that signal for a fraction of the cost and time it takes to build in order to find out.

How this connects to the module. This is the module's hinge lesson: it reuses timeToEvidence() from lesson 4 with no changes —the same function, the same buildAndSee— and adds the alternative path, discover. The result of this comparison is the full argument that sustains the rest of the guide: if discovering costs a fraction of building and arrives sooner, then learning to discover well (modules 2 through 7) is the most profitable investment a team can make before committing to build.

An analogy: the tasting before you buy the full case

Before buying a full case of wine for an event, almost nobody buys blind: they ask for a glass, taste it, and decide. The glass doesn't tell you absolutely everything you're going to experience with the full case —a single glass doesn't capture what it feels like to serve twelve bottles at a real party—, but it tells you the essential thing: whether you like the wine or not, well before paying for the entire case. Buying the full case "to try it" would be absurd — nobody does that, and yet that's exactly what a team does when it builds the full feature to find out if users are going to like it.

Discovery is a product bet's tasting glass: it doesn't replace the full experience of having the finished product, but it gives you the essential signal —does this work, or not?— for a fraction of the cost, before you commit to the full case.

Worked example: two paths, the same essential evidence

We pick timeToEvidence() back up exactly as it was left in lesson 4, unchanged, and add the second path: discover. This is the same cheap test the previous guide had already anticipated in its module 6 — showing hand-curated recommendations to 50 users for a week, and comparing their purchases against a control group that doesn't see them. Nobody builds the automatic engine yet; someone on the team picks the recommendations by hand, one by one.

// timeToEvidence: the same function from lesson 4, unchanged.
// Today we add the second path -- discover -- for the full comparison.
function timeToEvidence(path) {
  const costPersonDays = path.steps.reduce((sum, s) => sum + s.personDays, 0);
  const calendarDaysToEvidence = path.steps.reduce((sum, s) => sum + s.calendarDays, 0);
  return { name: path.name, costPersonDays, calendarDaysToEvidence, evidence: path.evidence };
}

const buildAndSee = {
  name: 'buildAndSee',
  evidence: 'full engine in production; measured with real purchases over weeks',
  steps: [
    { label: 'design the recommendations engine', personDays: 5, calendarDays: 5 },
    { label: 'build the basic engine (3 person-months)', personDays: 60, calendarDays: 60 },
    { label: 'QA and deploy to production', personDays: 5, calendarDays: 5 },
    { label: 'wait for enough purchase volume to measure', personDays: 0, calendarDays: 14 },
  ],
};

const discover = {
  name: 'discover',
  evidence: 'manual curation for 50 users, compared against a control group',
  steps: [
    { label: 'pick 50 users and hand-curate recommendations', personDays: 2, calendarDays: 2 },
    { label: 'run the test for 1 week and measure purchases vs. control group', personDays: 1, calendarDays: 7 },
  ],
};

console.log('=== timeToEvidence: two paths to the same evidence ===\n');
[buildAndSee, discover].forEach((path) => {
  const r = timeToEvidence(path);
  console.log(r.name + ':');
  console.log('  cost: ' + r.costPersonDays + ' person-days');
  console.log('  evidence in: ' + r.calendarDaysToEvidence + ' calendar days');
  console.log('  evidence: ' + r.evidence);
  console.log('');
});

const b = timeToEvidence(buildAndSee);
const d = timeToEvidence(discover);
const costSavings = Math.round((1 - d.costPersonDays / b.costPersonDays) * 100);
const timeSavings = Math.round((1 - d.calendarDaysToEvidence / b.calendarDaysToEvidence) * 100);
console.log('=== Comparison ===');
console.log('discover costs ' + costSavings + '% less than buildAndSee (' + d.costPersonDays + ' vs ' + b.costPersonDays + ' person-days)');
console.log('discover reaches evidence ' + timeSavings + '% faster than buildAndSee (' + d.calendarDaysToEvidence + ' vs ' + b.calendarDaysToEvidence + ' days)');

What to expect. When you run the file with Node, the output is exactly this:

=== timeToEvidence: two paths to the same evidence ===

buildAndSee:
  cost: 70 person-days
  evidence in: 84 calendar days
  evidence: full engine in production; measured with real purchases over weeks

discover:
  cost: 3 person-days
  evidence in: 9 calendar days
  evidence: manual curation for 50 users, compared against a control group

=== Comparison ===
discover costs 96% less than buildAndSee (3 vs 70 person-days)
discover reaches evidence 89% faster than buildAndSee (9 vs 84 days)

Read these two numbers slowly, because they're the whole module's central result: discover costs 96% less (3 person-days versus 70) and reaches evidence 89% faster (9 days versus 84). That's not a marginal improvement — it's a two-order-of-magnitude difference in cost, and nearly a tenfold difference in speed, to reach a signal about the same central question: do users buy more when they see recommendations?

⚠️ These numbers are a teaching case, built with reasonable estimates for the Mercado example — not a real measurement from any company. What matters isn't that "discovering always costs exactly 96% less": it's the magnitude of the pattern, which repeats consistently in real product teams' practice, and the decision logic behind the calculation (cost in personDays, time in calendarDays), which is what you'll apply again and again through the rest of this guide.

Why discover doesn't give "the same" evidence, and why that's fine

Notice the evidence column for each path: buildAndSee measures "real purchases over weeks" — the automatic engine, in production, with all of Mercado's real traffic. discover measures "manual curation for 50 users, compared against a control group" — a much smaller sample, with recommendations picked by hand by one person, not an algorithm. They're different kinds of evidence, and it would be a mistake to pretend they're interchangeable in every sense: discover doesn't tell you whether the automatic engine is going to work technically, or how it behaves with millions of products, or whether it scales. What it does tell you, with a smaller sample but a design that compares against a control group, is the question that actually matters before building anything: whether showing personalized recommendations —even hand-curated ones— changes purchase behavior. That's the full risky assumption, no more and no less. Discovery doesn't replace full measurement in production; it replaces the need to build everything just to get the first signal on whether it's worth continuing.

Common mistakes

Asking discovery to prove everything the finished product is going to prove. What happens: someone objects that the discover test "doesn't prove the automatic engine works technically" or "doesn't prove it scales to millions of users", and uses that objection to dismiss discovery entirely. Why it happens: it's true discovery doesn't prove those things — but those aren't the risky assumption being tested. The risky assumption is about purchase behavior, not engineering performance. How to spot it: the objection to a discovery test points at a question that test never promised to answer. How to fix it: before designing or criticizing a test, go back to the exact assumption being tested (in this case: "users buy more if they see personalized recommendations") — the detailed design of that correspondence, between assumption and test, is exactly module 4's topic.

Using the cost savings as an excuse to never build. What happens: in the opposite direction, a team falls so in love with how cheap discovery is that it starts treating it as a permanent substitute for building, running test after test without ever committing to build the real version, even once the evidence is already clear. Why it happens: discovery feels safe and low-risk, while building always means committing real resources — it's tempting to stay in the comfortable zone of "keep testing". How to spot it: the team has already run several tests with consistently positive signals about recommendations, and keeps postponing the decision to build the real engine. How to fix it: discovery exists to inform the decision to build, not to replace it indefinitely — this guide's module 7 (synthesizing and deciding) gives you the explicit criterion for knowing when the evidence is already enough and it's time to build.

Comparing discover's cost against the backlog's total cost, not against buildAndSee. What happens: someone argues that "3 person-days is nothing compared to everything the team has to do this quarter", minimizing the value of the savings by comparing it against the wrong magnitude. Why it happens: 3 person-days, in isolation, sounds like a small number, easy to dismiss as "not even worth measuring". How to spot it: the conversation about discovery's cost never compares it directly against the cost of the alternative it replaces. How to fix it: the right comparison, as you saw in today's example, isn't "3 days is little" — it's "3 days instead of 70 days, for the same question". The relevant savings is always against the path discovery avoids, not against the team's total workload.

Exercises

Exercise 1 — Recalculate with a more expensive discover. Without running Node, if the discover test needed twice the users (100 instead of 50) and that doubled its cost to 6 person-days while the time stayed at 9 calendar days, what would the new costSavings be against buildAndSee (70 person-days)? Is it still a huge difference?

See solution

costSavings = 1 - 6/70 = 0.914, rounded to 91% — still a huge difference, only 5 percentage points below the original 96%. This exercise shows something important: the module's conclusion (discovery costs a fraction of building) is robust even if the specific estimates change quite a bit. Doubling the discovery test's cost barely moves the conclusion, because the gap between 70 person-days and any single-digit number is still enormous in relative terms.

Exercise 2 — Explain which question each path answers. In a two-column table, write which question buildAndSee answers well that discover doesn't, and which question discover answers well that buildAndSee takes too long to answer.

See solution
buildAndSee answers welldiscover answers well
Does the automatic engine work technically at full scale?Do users change their purchase behavior when they see recommendations?
How does the system behave with millions of real products?Is it worth, in principle, investing in building the full engine?
What's the real, sustained impact in production, long term?What's the first, cheap, fast signal, before committing weeks of work?

The point of the table: they're not the same question, and that's why they don't compete — discover answers the question that needs answering first, before it makes sense to invest in answering the questions only buildAndSee can answer.

Exercise 3 — Argue with the numbers in front of a skeptic. A manager says: "I'd rather they build it directly — that way we have the real product, not an experiment with 50 people." Using today's exact numbers, write a 3-4 sentence response.

See solution

One possible answer: "I get the instinct, but look at the numbers: building the full engine costs 70 person-days and takes 84 days to give us any real evidence. The test with 50 users costs 3 person-days and gives us a clear signal on the same central question —whether recommendations change purchase behavior— in just 9 days. I'm not proposing we stick with the experiment forever: if the signal is good, we build the full engine with a lot more confidence than if we'd started blind. But risking 70 full days when we can get the same essential signal for 3 isn't caution, it's overspending." The argument works because it doesn't ask to give up on building — it asks to correctly sequence when to build, using the comparison's real numbers.

Summary and next step

In this lesson you ran the module's central comparison: discover —manual curation with 50 users against a control group— costs 96% less and reaches evidence 89% faster than buildAndSee —building the full engine—, for the same essential question about recommendations's risky assumption. You also saw why this comparison doesn't mean "discovery replaces building": it means discovery answers, first, the question that determines whether paying the cost of building is worth it or not.

Before moving on you should be able to: explain, with the exact numbers, how much cheaper and faster discover is than buildAndSee; name a question discover can't answer that needs buildAndSee; and defend, in front of a skeptic, why running the cheap test first isn't the same as avoiding building forever.

Lesson 6 answers the objection that almost always follows this comparison: "if discovery is so good, why don't we always do it, instead of just building directly — won't it slow us down?". There you'll see why discovery and delivery don't compete for the same time: they run in parallel, on two different tracks of the same team.

Resources

  • Teresa Torres, Continuous Discovery Habits — producttalk.org/continuous-discovery-habits. The book that develops, in much more detail than fits in this lesson, how a team designs and runs this kind of cheap test on a sustained basis. In English.
  • David J. Bland and Alexander Osterwalder, Testing Business Ideas summary — strategyzer.com/library/testing-business-ideas-book-summary. A catalog of dozens of cheap tests, of the same type as discover in this lesson, organized by what kind of evidence they give. In English.
  • Marty Cagan (SVPG), collection of articles on product discovery — svpg.com/insights/product-discovery-articles. A full archive of real cases where product teams got cheap evidence before building, with the same logic from this lesson applied to real products. In English.
← PreviousThe cost of building the wrong thingNext →Discovery and delivery, together