The limits of scoring

Overview

Lessons 3 through 6 gave you two real tools: RICE and ICE turn a subjective debate into a calculated, comparable, repeatable number. This lesson does something different: it teaches you where that number stops being reliable, not so you stop using it, but so you know exactly when to trust it and when not to.

The problem has a name from classic computing that fits perfectly here: garbage in, garbage out. The riceScore formula has no way of knowing whether a reach of 9000 comes from real analytics data or an optimistic hunch; it has no way of knowing whether a confidence of 1.0 is backed by an experiment or by the wish for your bet to win. The arithmetic is perfect —reach x impact x confidence / effort calculates exactly the same regardless of where those numbers came from—, and that perfection is exactly the problem: a flawless calculation on made-up data produces a made-up result with the appearance of precision.

How this connects to the module. Lessons 3 through 6 built the instrument; this lesson teaches you its limits, so you use it with your eyes open. The project (lesson 8) is going to show the same phenomenon with Mercado's full backlog: when a single input changes, the entire ranking can change, and that same sensitivity that makes the model useful (it lets you see the effect of new evidence) is what makes it vulnerable to manipulation (it lets you "win" by moving a number with no real evidence).

An analogy: the kitchen scale doesn't choose the recipe

Go back to the module introduction's scale. A kitchen scale precisely weighs whatever ingredients you put on it: 200 grams of flour is 200 grams, no matter what. But the scale doesn't decide what to cook, doesn't know whether the flour you put on it is right for the recipe, and has no way to detect if, instead of flour, you put sugar disguised as flour so the recipe "comes out better" on paper. It weighs whatever you give it, with absolute precision, and that precision says nothing about whether what you gave it was honest or correct.

riceScore is that scale. It calculates with absolute precision whatever you give it —reach x impact x confidence / effort, no rounding errors, no favoritism—, but it has no way to verify whether the confidence you gave it is real data or a wish disguised as data. The chef —the team, your judgment— is who decides what to weigh and what to do with the result. The scale helps; it doesn't cook.

Worked example: the function doesn't tell an honest confidence apart from an inflated one

Let's isolate the problem with two hypothetical bets —not Mercado's, so you see the phenomenon without the full backlog's complexity—. boringFix is "the boring but necessary fix", with a confidence measured from real data: 0.8. favoriteProject is "your favorite project", with the same reach, impact, and effort in the same order of magnitude —the only variable that changes across the three runs is the confidence you decide to declare for it—:

// riceScore does not tell an HONEST confidence apart from an INFLATED one: it just
// multiplies whatever number you give it. Two hypothetical bets, same reach/impact/effort
// order of magnitude, to isolate ONE single variable: the confidence you declare for
// your own project.
function riceScore({ reach, impact, confidence, effort }) {
  return (reach * impact * confidence) / effort;
}

// "The boring but necessary fix" - real, measured confidence: 0.8 (solid data).
const boringFix = { reach: 4000, impact: 1, confidence: 0.8, effort: 1 };
const boringScore = riceScore(boringFix);

// "Your favorite project" - reach/impact/effort are equally solid; the only thing
// that changes across the three runs is the confidence YOU decide to declare.
function favoriteProject(confidence) {
  return { reach: 3500, impact: 1, confidence, effort: 1 };
}

console.log('=== boringFix (measured confidence: 0.8) ===');
console.log('  score = ' + boringScore + '\n');

console.log('=== favoriteProject, varying ONLY the declared confidence ===\n');
for (const c of [0.5, 0.8, 1.0]) {
  const score = riceScore(favoriteProject(c));
  const winner = score > boringScore ? 'favoriteProject' : 'boringFix';
  const label = c === 0.8 ? '(same level as boringFix, honest)' : c === 1.0 ? '(inflated to "I am sure")' : '(honest, low)';
  console.log('  confidence=' + c + '  ' + label);
  console.log('  score=' + score + '  ->  wins: ' + winner + '\n');
}

console.log('The function does not know if 1.0 is real data or a wish. It just multiplies.');

What to expect. When you run the file with Node, the output is exactly this:

=== boringFix (measured confidence: 0.8) ===
  score = 3200

=== favoriteProject, varying ONLY the declared confidence ===

  confidence=0.5  (honest, low)
  score=1750  ->  wins: boringFix

  confidence=0.8  (same level as boringFix, honest)
  score=2800  ->  wins: boringFix

  confidence=1  (inflated to "I am sure")
  score=3500  ->  wins: favoriteProject

The function does not know if 1.0 is real data or a wish. It just multiplies.

Read the three runs carefully. With an honest, low confidence (0.5), favoriteProject clearly loses. With an honest confidence equal to boringFix's (0.8), favoriteProject still loses —its reach (3500) is a bit lower than boringFix's (4000), and at equal confidence, that's enough to lose—. But with confidence: 1 —inflated, with no new data justifying it, just the conviction of whoever's defending the project— favoriteProject wins. The function did exactly what it was asked: multiply. It did nothing wrong. The problem isn't in the arithmetic; it's in the input, and the arithmetic has no way to detect it.

This is "garbage in, garbage out" in its purest form: the same code, running perfectly, produced a defensible result in two runs and a manipulated one in the third, and the output alone doesn't tell you which is which. You only know if, for each number, you can answer "where did this come from?" with a real source —exactly the calibration discipline you built in lesson 5.

The score informs; it doesn't replace judgment

There's a second way scoring fails, subtler than inflating a number: using it after you've already decided, to justify a decision you made for other reasons. If the team has already decided, politically, that they're going to build X this quarter, and then they calculate its riceScore adjusting the inputs until the number "confirms" the decision, the calculation isn't informing anything —it's acting as an alibi—. The signal that this is happening: the numbers change after knowing the result someone wanted, not before.

The right way to use the score, instead, is exactly the opposite: you estimate each bet's four inputs before knowing which one "should" win, with the best available source for each, you calculate, and then you discuss the result —including the possibility that the number surprises everyone, as happened with fasterCheckout against improvedSearch in the introduction—. If the ranking never surprises you, be suspicious: it could be that your judgment was already excellent, or it could be that you're adjusting the inputs to confirm what you already believed.

Common mistakes

Inflating Confidence so your favorite bet wins. What happens: as you saw in today's example, someone raises their preferred project's confidence to 1.0 with no new data justifying it, specifically to beat another bet in the ranking. Why it happens: confidence is the most subjective of the four inputs and the easiest to move without anyone noticing right away. How to spot it: ask, bet by bet, "what changed for this number to go up?" —if the answer is "nothing, but I want it to win", there's the problem—. How to fix it: demand the source for any confidence of 0.8 or higher, following lesson 5's calibration table, before accepting any upward change.

Ignoring Reach and prioritizing whatever you like technically. What happens: someone argues for a bet by citing how interesting it would be to build, without mentioning how many people would use it nor how much it would move them —essentially, skipping the formula entirely and using the score only when it's convenient—. Why it happens: technical enthusiasm is genuine and feels like a sufficient reason. How to spot it: the defense of a bet includes none of RICE's four numbers, or includes them only after someone asks for them. How to fix it: as you saw in lesson 4, a big reach isn't optional in the argument; if your favorite bet doesn't have good numbers, the honest judgment is to accept it isn't the priority —not to stop calculating so you don't have to see it—.

Treating the score as absolute truth and switching off judgment. What happens: the team calculates the ranking once and treats it as a final, indisputable decision, without revisiting the inputs when new information appears, nor questioning a result that looks odd. Why it happens: a final number feels more objective —and more comfortable to defend— than continuing to discuss judgment. How to spot it: nobody looks at a bet's riceScore again after calculating it once, even if weeks have passed and new evidence is available. How to fix it: the score is a snapshot of your best estimate today; when the evidence changes —a new experiment, an analytics data point you didn't have—, recalculate. Lesson 8's project practices exactly this: recalculating the ranking when an input changes.

Exercises

Exercise 1 — Identify the warning sign. In a prioritization meeting, someone says: "let's change recommendations's confidence from 0.5 to 0.9, that way it ranks better." What's missing from that sentence for the change to be legitimate, per what you saw in this lesson and in lesson 5?

See solution

What's missing is the source: the sentence justifies the change by its effect on the ranking ("that way it ranks better"), not by new evidence about recommendations itself. A legitimate confidence change is justified with something like "we raised confidence from 0.5 to 0.8 because we ran a pilot experiment with simple recommendations and saw a clear improvement in clicks" —a reason that would exist independently of where the bet lands in the ranking—. If the only reason given is the final position, it's exactly this lesson's "using the score to justify what you already decided" pattern.

Exercise 2 — Recalculate with an honest confidence. Using favoriteProject's data from the worked example (reach: 3500, impact: 1, effort: 1), what minimum confidence would it need to exactly tie boringFix (score 3200)? Show the calculation.

See solution

You need to solve 3500 x 1 x confidence / 1 = 3200, that is, confidence = 3200 / 3500 ≈ 0.914. Since confidence in RICE doesn't have a fixed five-value scale like impact —it's a free fraction between 0 and 1—, that 0.914 (91.4%) would be the exact tying point. But notice the practical problem: 0.914 isn't a confidence level lesson 5's calibration table easily recognizes (the reference levels are 1.0, 0.8, 0.5); a confidence "hand-tuned" to such a specific number, just to tie or beat another bet, is itself a manipulation warning sign, regardless of whether the final number "sounds" reasonable.

Exercise 3 — Design an anti-manipulation rule. Based on what you saw in this lesson, propose a simple rule a team could adopt to reduce the risk of someone inflating a confidence with no evidence. (There's no single correct answer; evaluate your proposal against the worked example.)

See solution

A reasonable rule, common among teams that take RICE seriously: any confidence of 0.8 or higher must come, in writing, with the specific source backing it —a link to the experiment, the analytics data, or the research supporting it—. Without that written source, the default confidence is 0.5 or lower. Applied to the worked example: if someone proposes confidence: 1.0 for favoriteProject without being able to point to a concrete source, the rule automatically reverts it to 0.5, and at 0.5 the score is 1750, well below boringFix (3200) —the honest ranking is restored without anyone needing to "catch" the manipulation in the meeting; the rule prevents it before it happens—.

Summary and next step

In this lesson you put an honest limit on the tool you built in lessons 3 through 6: RICE and ICE calculate with absolute precision, but they cannot tell a real estimate apart from an inflated one —you saw, with favoriteProject, that the same reach/impact/effort and just one manipulated confidence are enough to flip the ranking, without the function doing anything "wrong"—. This is garbage in, garbage out, and the defense isn't to stop using the score: it's to demand a source for each input, especially for confidence, and to use the score before deciding, not afterward to justify what was already decided.

Before moving on you should be able to: explain "garbage in, garbage out" in your own words as applied to RICE; identify the signal that someone is adjusting an input to win the ranking instead of to reflect evidence; and tell "the score informs the decision" apart from "the score replaces judgment".

Lesson 8's project closes the module by bringing all of this together with Mercado's full backlog: you're going to score and order the five bets with prioritize, and then run a sensitivity analysis —changing a single input and seeing how much the ranking moves— to confirm, with real numbers, exactly how fragile or how robust a prioritization decision is depending on the quality of its estimates.

Resources

• ProductPlan, "How to Choose the Right Feature Prioritization Framework" — productplan.com/learn/how-to-choose-the-right-feature-prioritization-framework. An honest overview of scoring frameworks' limitations, including the subjectivity of estimating impact and confidence. In English.
• Melissa Perri, Escaping the Build Trap — on how even the most rigorous prioritization tools fail if used to justify decisions already made instead of to inform them; the same thesis from module 1 applied to scoring. In English.
• Marty Cagan / SVPG, Silicon Valley Product Group blog — svpg.com/articles — on why no framework replaces the judgment of a product team with good instincts; frameworks structure the conversation, they don't substitute for it. In English.