Module 6: Thinking In Bets And Assumptions

Risk × Impact

Next →

Overview

Lesson 4 left two assumptions tied in the highest-risk quadrant, with no way to break the tie: the importance/evidence grid, with only three levels per axis, doesn't have the resolution needed. This lesson replaces those levels with numbers: each assumption gets a risk —the estimated probability, between 0 and 1, that it's wrong— and an impact —a 1-to-5 scale of how much it damages the thesis if it is—. Multiplied, risk x impact gives a comparable score, and rankAssumptions() orders any list of assumptions from highest to lowest score, marking the first as the riskiest assumption.

Notice the formula's shape: it's deliberately the same logic you already used in module 3 with riceScore, applied to a different object. There you multiplied reach x impact x confidence / effort to order bets; here you multiply risk x impact to order the assumptions within a single bet. It's the same discipline —turning a subjective debate into an explainable number— applied one level deeper, inside the foundation.

How this connects to the module. Lesson 4 found, with a qualitative tool, two candidates for riskiest assumption. This lesson builds rankAssumptions(), runs it over recommendations's four assumptions, and breaks the tie with a number. Lesson 6 takes today's top-ranked assumption and decides how to test it; lesson 7 uses this same result to adjust the confidence you put on recommendations in module 3.

An analogy: how an insurer decides which risk worries it more

An insurance company doesn't decide how much to charge for a policy by asking only "how likely is something bad to happen?". A small kitchen fire is relatively likely but, if it happens, the damage is usually bounded. An earthquake is much less likely, but if it happens, the damage can be total. No serious insurer decides its exposure by looking only at probability, nor only at potential damage separately — they multiply the two: probability × severity = expected risk, and it's that number, not either one alone, that determines how much care —and how much capital— they dedicate to each type of risk.

risk x impact is exactly that same calculation, applied to a product assumption instead of a policy. An assumption with high risk but low impact —like the effort of building the recommendation engine, which is probably poorly estimated (risk high) but, if it is, only costs more time (impact low)— shouldn't worry you as much as an assumption with moderate risk but catastrophic impact. The multiplication captures that combination in a single number; looking at either variable alone, as lesson 4's grid did, can lead you to the wrong conclusion.

Worked example: rankAssumptions() over recommendations's four assumptions

We assign risk (0 to 1) and impact (1 to 5) to each of the four assumptions you separated in lesson 2, and build rankAssumptions():

// rankAssumptions(assumptions): each assumption has risk (probability, 0-1,
// that the assumption is false) and impact (1-5, how much it hurts the bet
// if it is). Sorts from highest to lowest risk*impact and marks the first
// as riskiest.
function rankAssumptions(assumptions) {
  return assumptions
    .map((a) => ({ ...a, score: a.risk * a.impact }))
    .sort((a, b) => b.score - a.score)
    .map((a, i) => ({ ...a, riskiest: i === 0 }));
}

// "recommendations"'s 4 assumptions (the ones classify() separated in lesson 2).
const assumptions = [
  { text: 'Users are going to buy more if they see personalized recommendations',
    risk: 0.6, impact: 5 },
  { text: 'The team can build a basic recommendation engine in 3 person-months',
    risk: 0.4, impact: 2 },
  { text: 'Sellers are not going to complain about their less popular products becoming less visible',
    risk: 0.3, impact: 3 },
  { text: 'Showing recommendations on the homepage does not noticeably slow down load speed',
    risk: 0.2, impact: 3 },
];

console.log('=== "recommendations"\'s assumptions, ordered by risk x impact ===\n');
const ranked = rankAssumptions(assumptions);
ranked.forEach((a, i) => {
  console.log(
    '  ' + (i + 1) + '. risk=' + a.risk + '  impact=' + a.impact +
    '  ->  score=' + a.score.toFixed(1) + (a.riskiest ? '   <-- RISKIEST ASSUMPTION' : '')
  );
  console.log('     ' + a.text);
});

console.log('\nThe riskiest assumption: "' + ranked[0].text + '"');
console.log('(score=' + ranked[0].score.toFixed(1) + ', the highest of the 4)');

What to expect. When you run the file with Node, the output is exactly this:

=== "recommendations"'s assumptions, ordered by risk x impact ===

  1. risk=0.6  impact=5  ->  score=3.0   <-- RISKIEST ASSUMPTION
     Users are going to buy more if they see personalized recommendations
  2. risk=0.3  impact=3  ->  score=0.9
     Sellers are not going to complain about their less popular products becoming less visible
  3. risk=0.4  impact=2  ->  score=0.8
     The team can build a basic recommendation engine in 3 person-months
  4. risk=0.2  impact=3  ->  score=0.6
     Showing recommendations on the homepage does not noticeably slow down load speed

The riskiest assumption: "Users are going to buy more if they see personalized recommendations"
(score=3.0, the highest of the 4)

Lesson 4's tie got resolved clearly: "users buy more when they see recommendations" wins with score=3.0, more than three times the score of "sellers are not going to complain" (0.9), the other candidate that had landed in the same high/low quadrant. The numbers explain why: even though both have relatively high impact (5 against 3), the first also has the highest risk of the four (0.6 — nobody has yet tested whether personalization actually changes purchase behavior at Mercado), while the second has a more moderate risk (0.3 — the team has some intuition, based on how sellers have reacted to previous visibility changes, that the reaction wouldn't be so negative).

Also notice the order of the other two: build effort (score=0.8) falls below "sellers" (score=0.9), despite having a higher risk (0.4 against 0.3) — because its impact is lower (2 against 3). This is exactly the same phenomenon you saw with RICE in module 3: no single variable decides on its own; it's the combination that orders things.

How risk and impact get calibrated (a pedagogical model, not a lab measurement)

Just like reach, impact, and confidence in RICE (module 3), risk and impact here are team estimates, not lab-precision measured data — always declare that when you use them. A practical guide for calibrating them with some discipline, instead of "by eye":

risk (0 to 1)   When to use it
─────────────  ──────────────────────────────────────────────────────
0.1 - 0.2       There is strong indirect evidence favoring that the
                assumption is true (data from a similar situation).
0.3 - 0.5       Uncertain: no clear data either for or against.
0.6 - 0.9       Little grounding; the team has no solid reason
                to believe it is true, beyond wishful thinking.

impact (1 to 5)  How much it damages the THESIS if the assumption is false
─────────────  ──────────────────────────────────────────────────────
1               Minor detail: gets adjusted with no effect on the result.
2               Costs more time or money, but the thesis still stands.
3               Damages a real part of the expected result.
4               Seriously calls into question whether the bet is worth it.
5               If false, the thesis completely loses its reason to exist.

With this table, "users buy more" gets impact: 5 because it's, literally, the mechanism by which the thesis would be true — without it, there's no other way the bet raises GMV. "Sellers don't complain" gets impact: 3, not 5, because even though it creates a real seller-relations problem, the thesis about GMV could, in principle, still be true even if some sellers protest.

Common mistakes

Confusing RICE's impact with rankAssumptions()'s impact. What happens: someone tries to reuse the impact RICE assigned to the full bet (a fixed scale of 3/2/1/0.5/0.25, how much it moves each person the bet reaches) as if it were the same number as an individual assumption's impact (how much it damages the thesis if that assumption is false). Why it happens: it's the same word, and it sounds reasonable to reuse the number you already calculated in module 3. How to spot it: someone asks "wasn't this bet's impact: 1 in RICE? why is it 5 now?" — confusing two completely different questions. How to fix it: clarify they're two scales with different purposes — RICE's impact measures how much the bet moves if it works; this lesson's impact measures how much it damages the thesis itself if a specific assumption turns out false. They share a name for vocabulary convenience, not because it's the same calculation.

Estimating risk and impact without being able to explain why, adjusting them so the assumption you already "suspected" wins. What happens: someone sets risk: 0.6 on the assumption they personally believe is most dangerous, with no explicit reason beyond "it seems that way to me" — the same pattern of inflating confidence you saw in module 3, now applied to risk and impact. Why it happens: they're the formula's two most subjective numbers, and it's tempting to adjust them so the ranking confirms a prior intuition instead of letting the ranking inform that intuition. How to spot it: nobody can answer "why this risk and not another?" with anything more than "it seems that way to me". How to fix it: demand, for any risk and impact of 0.6 or higher (or impact of 4 or 5), a concrete, written reason — just as you demanded a source for RICE's confidence in module 3. If the only reason is a hunch, the default number should be more conservative.

Using an impact scale that's too fine, faking a precision that doesn't exist. What happens: instead of this lesson's 1-to-5 scale, someone starts assigning values like impact: 3.7 or impact: 4.2, arguing over tenths-of-a-point differences between assumptions. Why it happens: a number with more decimals feels more rigorous, even though the estimate behind it remains, at bottom, a qualitative judgment. How to spot it: the team's discussion centers on whether something is 3.5 or 3.8, instead of whether it "damages a real part of the result" or "seriously calls the bet into question". How to fix it: as with module 3's RICE impact scale (deliberately five fixed values, not a continuum), use this lesson's table's five whole-number levels — the coarse scale forces you to decide with judgment, instead of faking an exactness the nature of the estimate can't support.

Exercises

Exercise 1 — Recalculate after a review. The team runs a quick survey of 15 active sellers and discovers most react worse than expected to the idea of their products becoming less visible. Without running Node, if "sellers are not going to complain"'s risk rises from 0.3 to 0.7 (impact stays at 3), does the ranking's order change? Show the calculation.

See solution

New score: 0.7 x 3 = 2.1. Compared to the original ranking (personalization 3.0, sellers 0.9 → now 2.1, effort 0.8, performance 0.6), "sellers are not going to complain" moves from spot 2 to spot 2 —it stays in the same place, since it was already second—, but now with a score much closer to the riskiest assumption's (2.1 against 3.0, instead of 0.9 against 3.0). The order doesn't change, but the distance does: the survey turned "sellers are not going to complain" into a serious candidate for riskiest assumption, though it still doesn't beat "users buy more". This is exactly the kind of new evidence that, in a different case, could flip the ranking — as you saw with RICE's sensitivity in module 3.

Exercise 2 — Design a fifth assumption and place it on the ranking. Go back to lesson 4's exercise 2 assumption: "Mercado's catalog has enough product variety per category that recommendations don't always repeat the same thing". Assign it a reasonable risk and impact (with your own justification) and calculate where it would land in the five-assumption ranking.

See solution

A reasonable assignment: risk: 0.4 (nobody has measured it yet, but there's no strong signal the catalog is scarce either) and impact: 4 (if the catalog lacks variety, recommendations feel repetitive and probably don't change purchase behavior as expected — it's very close to sinking the thesis's central mechanism, even if it isn't the mechanism itself). Score: 0.4 x 4 = 1.6. Compared to the original four's ranking (3.0, 0.9, 0.8, 0.6), this new assumption would land in second place, between "users buy more" (3.0) and "sellers are not going to complain" (0.9) — a real candidate for second test priority, even though the riskiest assumption stays the same.

Exercise 3 — Spot the manipulation. A teammate, an enthusiastic defender of recommendations, proposes lowering "users are going to buy more"'s risk from 0.6 to 0.2, arguing "I'm sure it's going to work, I've seen recommendations work everywhere." Is that a valid argument per this lesson's calibration table? What would you ask them?

See solution

It isn't a valid argument. "I've seen it work everywhere" is, at best, evidence from a similar situation —other platforms, not Mercado—, which on RICE's calibration table (module 3) would correspond to a confidence of 0.8, not the certainty a risk: 0.2 would imply here. It's also exactly the same "inflating confidence so your favorite bet wins" pattern from module 3, now applied to risk. Before accepting the change, you'd ask: "do we have any of Mercado's own behavior data specifically backing that personalization changes purchasing, or is it just the generalization that 'recommendations work in the industry'?". Without Mercado's own data, the correct risk stays high — 0.6 or more—, not 0.2.

Summary and next step

In this lesson you built rankAssumptions() and resolved the tie lesson 4 left behind: among recommendations's four assumptions, the riskiest —with score=3.0, more than three times the next— is "users are going to buy more if they see personalized recommendations". You also learned to calibrate risk and impact with an explicit table, instead of "by eye", and you saw that confusing this impact scale with RICE's is an easy mistake to make if you don't carefully distinguish which question each one answers.

Before moving on you should be able to: calculate risk x impact for any assumption; explain why the multiplication captures something neither variable alone captures; and tell RICE's impact apart from this lesson's impact.

Lesson 6 takes today's riskiest assumption and answers the next mandatory question: how do you test it, without building the full recommendation engine first? And it connects directly to module 5's MVP — the right experiment isn't "the easiest to put together", it's the one that specifically attacks this assumption.

Resources

← PreviousThe riskiest assumptionNext →Test the riskiest first