Module 6: Thinking In Bets And Assumptions

The riskiest assumption

Next →

Overview

recommendations rests on four untested assumptions. This lesson's question isn't "which of the four raises the most doubt for me?" nor "which is hardest to estimate?" — it's a more precise question: if this specific assumption turns out false, does the bet's entire thesis collapse, or can the team move forward adjusting something? The assumption that answers "everything collapses" is the riskiest assumption: the one that concentrates, at the same time, the highest probability of being wrong (risk) and the greatest damage if it is (impact). It isn't necessarily the one that generates the most discussion at the meeting, nor the one that makes the team most uncomfortable — it's often an assumption so taken for granted that nobody names it out loud.

How this connects to the module. Lesson 3 gave you the vocabulary —thesis, assumption stack— and counted how many assumptions support each bet. This lesson goes into that stack and searches, among recommendations's four assumptions, for the one that literally holds up the thesis's weight. Lesson 5 is going to put an exact number on that search; today you do it with a simpler tool, so you see the criterion first, before the calculation.

An analogy: the block tower and the block you can't remove

Think about a tower built with wooden blocks, the kind used in the game where each player removes a block without the tower falling. A tower like that has dozens of blocks, and most can be removed with no problem — the structure tolerates it, resettles, stays standing. But there are some blocks —usually few, sometimes just one— that hold up the weight of several entire levels. Remove any of those, and the whole tower collapses, no matter how carefully you placed the rest.

What's interesting —and what makes this game hard— is that it isn't always obvious, just by looking, which block is the load-bearing one. A block that looks central might not be supporting anything; one that looks marginal might be exactly the one holding up three levels above it. The only reliable way to know is a specific question for each block: "if I remove this one, what happens above it?" — not "how nervous does removing this make me?", which is a completely different question and, often, misleading.

recommendations's four assumptions are that tower's blocks. The question the next few pages teach you to ask, for each one, is exactly that: if this assumption is false, what happens to the thesis? Not "how sure am I of this?" — that confuses personal discomfort with real structural risk.

Worked example: placing the four assumptions on the importance/evidence grid

David Bland and Alexander Osterwalder, in Testing Business Ideas, propose a tool called assumptions mapping: placing each assumption on a grid with two axes —importance (how much does this assumption matter to the thesis?) and evidence (how much real evidence already exists about it?)—. The quadrant of high importance and low evidence is what they call the "leap of faith": the zone worth looking at first. Let's place recommendations's four assumptions there, still without putting an exact number on it —that's lesson 5—, just with a first qualitative read:

// mapToQuadrant: places an assumption on David Bland's grid (Testing Business
// Ideas): importance (how much it matters to the thesis) x evidence (how much
// real evidence already exists). The "high importance + low evidence" quadrant
// is the "leap of faith": the candidate for riskiest assumption. This is a
// FIRST pass, still without numbers (that's lesson 5).
function mapToQuadrant({ importance, evidence }) {
  const highImportance = importance === 'high';
  const lowEvidence = evidence === 'low';
  if (highImportance && lowEvidence) return 'LEAP OF FAITH (test first)';
  if (highImportance && !lowEvidence) return 'important, but already has evidence';
  if (!highImportance && lowEvidence) return 'no evidence, but does not sink the thesis';
  return 'safe zone';
}

const assumptions = [
  { text: 'Users are going to buy more if they see personalized recommendations',
    importance: 'high', evidence: 'low' },
  { text: 'The team can build a basic engine in 3 person-months',
    importance: 'low', evidence: 'low' },
  { text: 'Sellers are not going to complain about less visible products',
    importance: 'high', evidence: 'low' },
  { text: 'Showing recommendations does not slow down load speed',
    importance: 'low', evidence: 'medium' },
];

console.log('=== "recommendations"\'s assumptions, placed on the importance x evidence grid ===\n');
const mapped = assumptions.map((a) => ({ ...a, quadrant: mapToQuadrant(a) }));
mapped.forEach((a) => {
  console.log('  [' + a.quadrant + ']');
  console.log('  ' + a.text + '\n');
});

const leapsOfFaith = mapped.filter((a) => a.quadrant.startsWith('LEAP'));
console.log('Assumptions in "leap of faith": ' + leapsOfFaith.length);
leapsOfFaith.forEach((a) => console.log('  - ' + a.text));
console.log('\nThe grid found ' + leapsOfFaith.length + ' candidates, but does not say WHICH of the ' +
  leapsOfFaith.length + ' to test first: "high" and "high" is a tie. A number is needed.');

What to expect. When you run the file with Node, the output is exactly this:

=== "recommendations"'s assumptions, placed on the importance x evidence grid ===

  [LEAP OF FAITH (test first)]
  Users are going to buy more if they see personalized recommendations

  [no evidence, but does not sink the thesis]
  The team can build a basic engine in 3 person-months

  [LEAP OF FAITH (test first)]
  Sellers are not going to complain about less visible products

  [safe zone]
  Showing recommendations does not slow down load speed

Assumptions in "leap of faith": 2
  - Users are going to buy more if they see personalized recommendations
  - Sellers are not going to complain about less visible products

The grid found 2 candidates, but does not say WHICH of the 2 to test first: "high" and "high" is a tie. A number is needed.

Look at what the grid achieved and what it still can't do. It did achieve immediately ruling out two assumptions —the effort of building the engine and the performance risk— as candidates for "the riskiest": neither, if it turns out false, sinks the thesis that "recommendations raise GMV"; in the worst case, they cost more time or require a technical adjustment, but the bet can still stand. It didn't achieve, on the other hand, deciding between the two that really do matter: both "users buy more" and "sellers don't complain" ended up in the same quadrant, high/low, tied. And they're very different assumptions in severity — if users don't buy more, the whole thesis loses its reason for existing; if sellers do complain, there's a real relationship problem with them, but the thesis that recommendations move GMV can still be true. The grid, with only three levels per axis (high/medium/low), doesn't have the resolution to separate them. That's exactly the limit that motivates lesson 5: moving from a three-level scale to a calculation with real numbers.

Marty Cagan's four risks, as a lens for finding candidates

Before mapping an assumption, it helps to know where to look. Marty Cagan, in his work with Silicon Valley Product Group, describes four types of risk any product bet can have:

Risk type              Question it answers                       Example in "recommendations"
──────────────────────  ────────────────────────────────────────  ─────────────────────────────────────
Value                   Are users going to want/use this?          "users buy more when they see
                                                                    recommendations"
Usability                Are users going to know how to use it?     (does not appear as its own
                                                                    assumption in this bet; assumed simple)
Feasibility              Can the team build it with the time and    "the team can build the basic
                          technology available?                     engine in 3 person-months"
Business viability        Does it work for the business -legal,     "sellers are not going to complain
                          sales, brand, other teams-?                about less visible products"

Cagan observes something important, and one that repeats in the recommendations example: engineering teams tend to feel more comfortable discussing feasibility risk —it's their technical territory, they have vocabulary and data to talk about it—, and tend to underestimate value risk — it's more uncomfortable, because it depends on human behavior that nobody in the room can control nor predict with the same precision as an effort estimate. That discomfort is, often, the exact reason the value assumption —"users are going to buy more"— ends up being the risky one nobody put first in the conversation, even though it's the one that decides whether the whole bet makes sense.

Common mistakes

Confusing "the assumption that generates the most discussion" with "the riskiest". What happens: the planning meeting spends most of its time debating whether 3 person-months is a realistic estimate for building the engine —a comfortable technical discussion, with data and precedents—, and almost no time on whether users are really going to buy more. Why it happens: technical discussions have shared language and clear criteria to settle them; discussions about human behavior don't, and that makes them uncomfortable to sustain. How to spot it: review how much of the meeting's time was spent on each assumption — if the time doesn't correlate with today's grid's high importance / low evidence quadrant, the team is debating what's comfortable, not what's risky. How to fix it: explicitly use this lesson's question —"if this is false, does the thesis collapse?"— for each assumption, and prioritize the conversation based on that answer, not on how easy it is to discuss.

Choosing as riskiest the one that's easiest to understand, not the one that sinks the thesis. What happens: someone proposes the riskiest assumption is "showing recommendations doesn't slow down load speed", because it's concrete, measurable, and the team knows exactly how to test it (a load test). Why it happens: an assumption easy to test feels, by association, like "the important one" — there's a bias toward what's immediately actionable. How to spot it: the assumption chosen as riskiest turns out to be, coincidentally, the easiest to validate with the tools the team already has on hand. How to fix it: as today's grid showed, the performance risk falls in "safe zone" — it has reasonable evidence (medium) and low structural importance. How easy something is to test is a lesson 6 consideration (test cheaply), not a criterion for deciding what's riskiest — they're two different questions, and in that order.

Treating a large reach or effort as synonymous with large risk. What happens: the team assumes the assumption about build effort (3 person-months, the "biggest", most visible figure of the four) must be the riskiest, simply because it's a number that draws attention. Why it happens: a large, explicit number feels more "important" than a qualitative claim about human behavior, even though they measure completely different things. How to spot it: the choice of the risky assumption gets justified by citing a magnitude (person-months, reach, effort) instead of answering "does this sink the thesis if it's false?". How to fix it: a number's size isn't the criterion — this lesson's grid explicitly classified the build effort as importance: low, because even if it's poorly estimated, the thesis that "recommendations raise GMV" can still be true; only the cost and time to find out changes, not whether it's worth trying.

Exercises

Exercise 1 — Apply the load-bearing block question. For each of recommendations's four assumptions, answer in one sentence: "if this assumption is false, does the thesis that recommendations raise GMV hold or collapse?".

See solution
  • "Users buy more when they see recommendations": it collapses. It's literally the mechanism by which the thesis would be true.
  • "The team can build the engine in 3 person-months": it holds, with more cost or time — the thesis about the GMV effect doesn't depend on how long it takes to build.
  • "Sellers are not going to complain": it holds as far as GMV goes, though it creates a different problem (seller relations) worth addressing separately.
  • "Showing recommendations doesn't slow down load speed": it holds, as long as the speed impact is minor — if it were catastrophic, it could sink the overall experience, but as an isolated assumption its severity is low.

Only the first answers "it collapses" with no conditions — it's the strongest candidate for riskiest assumption, confirming what the grid already hinted at.

Exercise 2 — Place a new assumption on the grid. The team adds a fifth assumption to recommendations: "Mercado's catalog has enough product variety per category that recommendations don't always repeat the same thing". With no data yet, what importance and evidence would you assign it, and which quadrant does it fall into?

See solution

A reasonable assignment: importance: high —if the catalog doesn't have enough variety, recommendations will always show the same few products, which probably doesn't change purchase behavior as much as expected, getting close to sinking the thesis— and evidence: low —nobody has measured yet how many distinct products there are, on average, per category—. That places it in LEAP OF FAITH, tied with the other two high-importance assumptions. This exercise shows something real: the more carefully assumptions get named, the more candidates show up in the risk quadrant — one more reason to need lesson 5's exact number, not just the qualitative placement.

Exercise 3 — Classify by Cagan's risk type. Using Cagan's table of four risks, classify this assumption from the improvedSearch bet: "the new search algorithm doesn't require rewriting the entire index, which would take 6 months". What type of risk is it, and why does that suggest it probably isn't that bet's riskiest assumption?

See solution

It's a feasibility risk: it asks whether the team can build it with the time and technology available, not whether users are going to care about the result. Following the same reasoning as the recommendations example, a pure feasibility risk —though real and costly if it materializes (6 months is a lot of time)— usually doesn't sink the thesis that "improving search raises conversion": if the algorithm takes longer than expected, the team loses time and money, but the question of whether improved search really changes purchase behavior —almost certainly a value risk— remains unanswered and remains more dangerous to the bet itself.

Summary and next step

In this lesson you learned to tell apart the assumption that holds up the thesis's weight from the ones that simply generate discomfort or require effort. The key question —"if this is false, does the thesis collapse or hold?"— and David Bland's importance/evidence grid took you from four assumptions to two strong candidates ("users buy more" and "sellers don't complain"), but you also saw the limit of a purely qualitative tool: when two assumptions fall in the same quadrant, the grid can't break the tie.

Before moving on you should be able to: apply the load-bearing block question to any assumption; place an assumption on the importance/evidence grid; and recognize, using Cagan's four risks, why value risk tends to be the most underestimated by engineering teams.

Lesson 5 resolves exactly the tie left pending: it turns importance and evidence into real numbers —risk and impact— and uses rankAssumptions() to order the four assumptions with no ambiguity, with the riskiest assumption on top.

Resources

  • David J. Bland, "How Assumptions Mapping Can Focus Your Teams On Running Experiments That Matter" — strategyzer.com/library/how-assumptions-mapping-can-focus-your-teams-on-running-experiments-that-matter. The article describing the importance/evidence grid used in this lesson, with the "leap of faith" quadrant explained in depth. In English.
  • Marty Cagan / SVPG, "The Four Big Risks" — svpg.com/four-big-risks. The source of the four risk types table (value, usability, feasibility, business viability) used to place where the riskiest assumption might be hiding. In English.
  • Marty Cagan / SVPG, "The Biggest Risk" — svpg.com/the-biggest-risk. On why the risk that nobody cares about the solution —value risk— tends to be the biggest and the most postponed by teams. In English.
  • Teresa Torres, "Opportunity Solution Trees" — producttalk.org/opportunity-solution-trees. On how to visually map the assumptions behind a solution to make visible which ones truly support the desired result. In English.
← PreviousA decision is a betNext →Risk × Impact