The cheapest experiment that reduces uncertainty

Overview

You already saw compareApproaches run once, in the module introduction, over the recommendations bet. This lesson opens it up from the inside: what each field measures, why effortRatio isn't the most important of the two numbers it produces, and —above all— why "the cheapest experiment" doesn't simply mean "the fastest experiment to build". You're going to apply the same model to a second full Mercado bet, to confirm the pattern isn't a recommendations coincidence: it repeats.

How this connects to the module. Lesson 2 gave you the criterion for knowing whether a candidate is a real MVP; lesson 3 taught you to tell what can be cut for free (scope) apart from what can't (learning). This lesson brings both together into a single question with a number: of all the plans that meet a real MVP's conditions and protect learning, which is cheapest? And it gives you the vocabulary to defend that choice with data, not intuition.

An analogy: the exploratory core sample, not the whole mine

A geologist who suspects there's gold in a plot of land doesn't start by digging the entire mine. They drill a core sample: a thin tube that extracts a column of earth a few centimeters in diameter, across the whole area they want to explore. That sample, analyzed in a lab, tells them with fair confidence whether there's enough mineral to justify the investment of opening the mine —at a tiny fraction of the cost of excavating—. If the sample comes back empty, they saved months of excavation. If it comes back promising, they now have real evidence, not a hunch, to justify the big investment.

What's interesting about the core sample, for this lesson, is that it isn't a smaller mine: it's a different instrument, designed specifically to answer "is there anything here?" at the lowest possible cost, without building a single meter of real tunnel. The cheapest experiment almost never looks, in its shape, like a reduced version of the final product —it looks like a measurement instrument designed on purpose for that one question—.

Worked example: compareApproaches in depth, with a second bet

Go back to the function you already used in the introduction:

function compareApproaches(bet) {
  const { name, riskiestAssumption, buildEverything, mvp } = bet;
  const effortRatio = Math.round((mvp.effort / buildEverything.effort) * 100);
  const sameEvidence = buildEverything.provesAssumption === mvp.provesAssumption;
  return { bet: name, riskiestAssumption, buildEverything, mvp, effortRatio, sameEvidence };
}

It has five pieces, and it's worth naming them one by one because each answers a different question:

• riskiestAssumption: the sentence that, if false, sinks the entire bet. It's the same for both plans —that's exactly what makes them comparable—.
• buildEverything.effort and mvp.effort: how much human work each plan consumes, in the same unit (person-days here).
• buildEverything.evidenceAt and mvp.evidenceAt: the moment, not just the cost, at which each plan produces evidence about the assumption. This field matters as much as effort, and gets forgotten often: two plans with the same total cost can differ enormously in when they deliver the answer.
• provesAssumption (inside each plan): whether that plan, once completed, actually puts riskiestAssumption to the test —not a similar assumption, the same one—.
• effortRatio and sameEvidence: the two numbers that summarize the comparison. effortRatio is how much the MVP costs as a percentage of building everything. sameEvidence is the check that both plans, despite the different cost, test the same thing.

Now apply it to a second bet from Mercado's backlog: Seller analytics dashboard, the one you rewrote in module 1's project with the mechanism "sellers see which products are moving and adjust price or stock in time".

const recommendationsBet = {
  name: 'Product recommendations carousel',
  riskiestAssumption: 'showing complementary products in the cart makes more buyers add a second product before paying',
  buildEverything: { plan: 'recommendation engine with collaborative filtering and its own microservice', effort: 40, evidenceAt: 'day 40', provesAssumption: true },
  mvp: { plan: 'fixed "bought together" table for the 20 highest-volume categories', effort: 4, evidenceAt: 'day 4', provesAssumption: true },
};

const sellerToolsBet = {
  name: 'Seller analytics dashboard',
  riskiestAssumption: 'if sellers see that products are moving slowly or running out of stock, they adjust price or stock in time, and that reduces lost sales',
  buildEverything: {
    plan: 'full dashboard with real-time charts, automatic alerts, and export, for the 3000+ sellers',
    effort: 25,
    evidenceAt: 'day 25',
    provesAssumption: true,
  },
  mvp: {
    plan: 'weekly email report to 15 pilot sellers with their 5 worst-rotating products, generated by hand with a query',
    effort: 3,
    evidenceAt: 'day 3',
    provesAssumption: true,
  },
};

console.log('=== compareApproaches: Seller analytics dashboard ===\n');
const r = compareApproaches(sellerToolsBet);
console.log('Riskiest assumption: ' + r.riskiestAssumption + '\n');
console.log('-- buildEverything --');
console.log('  plan: ' + r.buildEverything.plan);
console.log('  effort: ' + r.buildEverything.effort + ' person-days, evidence: ' + r.buildEverything.evidenceAt + '\n');
console.log('-- mvp --');
console.log('  plan: ' + r.mvp.plan);
console.log('  effort: ' + r.mvp.effort + ' person-days, evidence: ' + r.mvp.evidenceAt + '\n');
console.log('effortRatio: ' + r.effortRatio + '%, same key evidence: ' + r.sameEvidence);

console.log('\n=== The pattern repeats across both bets ===\n');
[recommendationsBet, sellerToolsBet].forEach((b) => {
  const res = compareApproaches(b);
  console.log(res.bet + ': the mvp costs ' + res.effortRatio + '% of the effort of building everything, same evidence: ' + res.sameEvidence);
});

What to expect. When you run the file with Node, the output is exactly this:

=== compareApproaches: Seller analytics dashboard ===

Riskiest assumption: if sellers see that products are moving slowly or running out of stock, they adjust price or stock in time, and that reduces lost sales

-- buildEverything --
  plan: full dashboard with real-time charts, automatic alerts, and export, for the 3000+ sellers
  effort: 25 person-days, evidence: day 25

-- mvp --
  plan: weekly email report to 15 pilot sellers with their 5 worst-rotating products, generated by hand with a query
  effort: 3 person-days, evidence: day 3

effortRatio: 12%, same key evidence: true

=== The pattern repeats across both bets ===

Product recommendations carousel: the mvp costs 10% of the effort of building everything, same evidence: true
Seller analytics dashboard: the mvp costs 12% of the effort of building everything, same evidence: true

The seller dashboard confirms recommendations wasn't a special case: 12% of the effort, same evidence, this time using something as simple as a weekly email generated by hand with a database query —not a single line of dashboard, not a chart, not an automatic alert—. The email report isn't "a simpler dashboard": it is, like the geologist's core sample, a different instrument, designed specifically to answer whether sellers act when they see the information, without building any of the visual infrastructure the full dashboard would need.

Why evidenceAt matters as much as effort

There's an easy trap here: focusing only on effortRatio and assuming "cheap" is the only thing that matters. But look at the difference between sellerTools's two plans: it isn't just that the MVP costs less (3 against 25 person-days) — it's that the evidence arrives 22 days sooner. Those 22 days aren't free: they're 22 days in which the team can be iterating on what it learned, instead of waiting for the full dashboard to finish to find out whether the assumption was true. A plan that costs little but takes almost as long to deliver evidence as building everything —imagine an MVP with effort: 3 but evidenceAt: day 24, for example, if it depended on a slow approval process— is a worse MVP than one with the same cost but immediate evidence. Both numbers, cost and moment, are part of "the cheapest experiment"; optimizing only one of the two is half a victory.

Common mistakes

Measuring "cheapest" only by cost, without verifying it tests the SAME assumption. What happens: the team finds an experiment with a very low effort and adopts it without verifying its provesAssumption really matches building everything's —it ends up testing a diluted or different version of the original question. Why it happens: a low effort number is easy to see and compare; verifying the assumption is exactly the same requires carefully reading both plans' riskiestAssumption, not just looking at the cost. How to spot it: explicitly ask "if this experiment goes well, does it confirm exactly the original assumption, or something similar but different?". How to fix it: sameEvidence isn't a decorative model detail — it's the mandatory check before accepting any low effortRatio as a win.

Ignoring evidenceAt and settling for just the effortRatio. What happens: as explained above, the team celebrates a low effortRatio without noticing when the evidence actually arrives —and discovers "cheap" didn't mean "fast"—. Why it happens: a single number (the savings percentage) is easier to repeat in a meeting than two numbers together. How to spot it: nobody in the meeting can say, without looking at the data again, what day the MVP's result is expected. How to fix it: always report both numbers together —how much it costs and when the answer arrives—, as compareApproaches does. Low cost without evidence speed is only half the victory.

The eternal MVP that never measures anything. What happens: the team builds the cheap experiment —the weekly report, the fixed table—, launches it, and never checks again whether it actually proved or disproved the assumption; the "experiment" keeps running indefinitely with nobody collecting the result. Why it happens: building the MVP feels like "we already did the hard part"; reviewing the results and deciding based on them is an extra step, easy to postpone one more week, and then another. How to spot it: the weekly seller report has been going out for two months and nobody can yet say whether sellers who receive it adjust their stock more than those who don't. How to fix it: an MVP with no date or closing criterion isn't an experiment, it's a half-built product nobody decided to maintain. Before launching it, define when and how the result gets reviewed — lesson 7 revisits this anti-pattern in more detail.

Exercises

Exercise 1 — Apply the model to a third bet. Mercado's team also evaluates Improved search: building a full ML ranking with synonyms, typo tolerance, and personalization costs effort: 35, with evidence on day 35. The MVP —a simple boost that prioritizes exact title matches for the 50 most-searched terms— costs effort: 5, with evidence on day 5. Both test the same assumption (provesAssumption: true in both). Run compareApproaches with this data and report the effortRatio.

{ bet: 'Improved search', effortRatio: 14, sameEvidence: true }

Exercise 2 — The cost of an ignored evidenceAt. Imagine two hypothetical MVPs for the same bet, both with effort: 5 (the exact same cost). The first has evidenceAt: 'day 5'; the second, because it depends on a slow legal approval process, has evidenceAt: 'day 30'. Without using effortRatio (which would be identical for both), explain in 2-3 sentences why the second is a worse MVP, even though it costs exactly the same.

Exercise 3 — Find the instrument, not the small mine. For exercise 1's improvedSearch bet, someone proposes as "MVP" building the same ML ranking but only for the electronics category, instead of the whole catalog (estimated effort: 30, barely less than building everything's effort: 35). Using the geological core sample analogy, explain why this isn't really "the cheapest experiment" even though it's technically smaller than the full version.

Summary and next step

In this lesson you opened up compareApproaches from the inside: you saw it compares two things at once —how much it costs (effort) and when the answer arrives (evidenceAt)—, and both matter, not just the first. You confirmed, with Seller analytics dashboard, that the 10-15% effort pattern with the same evidence isn't a recommendations coincidence: it repeats when the experiment is designed as its own instrument for the smallest question, not as a miniature of the full solution —the geological core sample, not a shorter mine—.

Before moving on you should be able to: name compareApproaches's five fields and what each one measures; explain why evidenceAt matters as much as effort; and recognize when a proposed "MVP" is still, at bottom, a reduced version of the final solution instead of a purpose-built instrument.

Lesson 5 changes axis: so far you saw how much the experiment costs and when it arrives; lesson 5 looks at its shape —why an experiment that touches every layer of the system, even if each is minimal, tests more than a complete but isolated feature.

Resources

• Eric Ries, The Lean Startup — theleanstartup.com. The concept of "validated learning per unit of time and effort" is this lesson's quantitative foundation: the right experiment doesn't just minimize cost, it minimizes time until learning. In English.
• Marty Cagan (Silicon Valley Product Group), "MVP" — svpg.com/articles. On the MVP seen as an experiment designed to reduce product, business, or viability risk —not as a scaled-down build—. In English.
• Intercom, "RICE: Simple prioritization for product managers" — intercom.com/blog/rice-simple-prioritization-for-product-managers. Though it's about prioritization (module 3), it's worth rereading the effort section with this lesson in mind: the same care in estimating cost applies to the experiment's cost, not just the full bet's. In English.