Stay honest: the final checklist

Overview

So far you've built two separate executable tools: sampleBias() (lesson 4) checks whether the sample you collected is biased toward one segment or missing a relevant one; rankByStrength() (lesson 6) weighs each piece of evidence by its type, no matter how convincing it felt to hear it. It's tempting to think that, with both tools in hand, you're already protected from confirmation bias. But the previous lesson's exercise 3 already hinted at the problem: a piece of evidence can have the maximum possible strength and still be unreliable, if it comes from a biased sample — and a perfectly representative sample can produce weak evidence, if nobody observed real behavior. Neither tool, on its own, tells the whole story.

This lesson adds no new model: it brings the two you already have together into a two-question honesty checklist, and runs it over two versions of the same recommendations discovery round — the rushed round, typical of a team without this module's tools, and the corrected round, after applying what was learned in lessons 2 through 6. The result isn't just an academic exercise: it's, literally, the checklist you're going to run in lesson 8's mini-project before handing any finding off to module 7 for synthesis.

How this connects to the module. This is the module's hinge lesson: it reuses sampleBias() (lesson 4) and rankByStrength() (lesson 6) exactly as they stood, with no changes at all, and runs them together inside honestyChecklist(), a small model that combines both results into a single verdict. What comes out of this checklist is exactly the input module 7 (synthesize(), updateConfidence()) needs to receive: evidence already verified as unbiased at its origin and strong enough in its type, not raw, unaudited evidence.

An analogy: the double review before signing

An important contract doesn't get signed after a single person reads it once and says "looks fine." It typically goes through two distinct reviews: someone checks whether the terms are fair (the content), and someone else checks whether the signatures and the parties involved are correct (the process). A contract with perfect terms, signed by the wrong person, is just as invalid as one with the right signature but abusive terms — neither review, on its own, is enough.

This lesson's checklist does exactly that double check on discovery evidence: sampleBias() checks "did the right person sign this?" —does the sample represent who it should represent?— and rankByStrength() checks "are the terms solid?" —is the evidence of the type that truly predicts future behavior?—. Passing only one of the two reviews isn't enough to trust the conclusion.

Worked example: honestyChecklist() over two discovery rounds

We compare recommendations's rushed round —lesson 4's biased sample, with evidence mostly of hypothetical and stated_opinion type— against a corrected round, with a more balanced sample and real behavior evidence.

// L7: sampleBias() (L4) and rankByStrength() (L6) reused unchanged,
// combined into a two-question honesty checklist, applied to two
// "recommendations" discovery rounds: the rushed round from before this
// module, and the corrected round after applying what was learned in L2-L6.
function sampleBias({ segments }) {
  const total = segments.reduce((sum, s) => sum + s.count, 0);
  const withShare = segments.map((s) => ({ ...s, share: +((s.count / total) * 100).toFixed(1) }));
  const dominant = withShare.reduce((max, s) => (s.share > max.share ? s : max));
  const missing = withShare.filter((s) => s.count === 0);
  const biased = dominant.share >= 70 || missing.length > 0;
  return { total, dominant, missing, biased };
}

const STRENGTH_BY_TYPE = { observed_behavior: 4, reported_behavior: 3, stated_opinion: 2, hypothetical: 1 };
function rankByStrength(evidences) {
  return evidences.map((e) => ({ ...e, strength: STRENGTH_BY_TYPE[e.type] })).sort((a, b) => b.strength - a.strength);
}

function honestyChecklist(round) {
  const sample = sampleBias(round.sample);
  const ranked = rankByStrength(round.evidence);
  const avgStrength = +(ranked.reduce((sum, e) => sum + e.strength, 0) / ranked.length).toFixed(2);
  const strongEnough = avgStrength >= 3;
  return {
    label: round.label,
    sampleBiased: sample.biased,
    dominantSegment: sample.dominant.segment + ' (' + sample.dominant.share + '%)',
    avgStrength,
    strongEnough,
    passes: !sample.biased && strongEnough,
  };
}

const rushedRound = {
  label: 'Rushed round (before this module)',
  sample: { segments: [
    { segment: 'power_user', count: 6 },
    { segment: 'friend_of_team', count: 1 },
    { segment: 'occasional_buyer', count: 1 },
    { segment: 'new_user', count: 0 },
  ]},
  evidence: [
    { claim: 'Said they would love to see recommendations', type: 'hypothetical' },
    { claim: 'Said they would probably buy more', type: 'hypothetical' },
    { claim: 'Found the mockup useful', type: 'stated_opinion' },
  ],
};

const correctedRound = {
  label: 'Corrected round (after L2-L6)',
  sample: { segments: [
    { segment: 'power_user', count: 3 },
    { segment: 'occasional_buyer', count: 3 },
    { segment: 'new_user', count: 2 },
  ]},
  evidence: [
    { claim: 'Clicked "recommended for you" on the fake door', type: 'observed_behavior' },
    { claim: 'Ignored the section entirely, zero clicks', type: 'observed_behavior' },
    { claim: 'Recounted a past purchase influenced by a recommendation', type: 'reported_behavior' },
  ],
};

console.log('=== honestyChecklist() over two "recommendations" discovery rounds ===\n');
[rushedRound, correctedRound].forEach((round) => {
  const result = honestyChecklist(round);
  console.log(result.label + ':');
  console.log('  biased sample: ' + result.sampleBiased + ' (dominant ' + result.dominantSegment + ')');
  console.log('  average evidence strength: ' + result.avgStrength + ' (>=3 required)');
  console.log('  passes the checklist: ' + result.passes);
  console.log('');
});

What to expect. When you run the file with Node, the output is exactly this:

=== honestyChecklist() over two "recommendations" discovery rounds ===

Rushed round (before this module):
  biased sample: true (dominant power_user (75%))
  average evidence strength: 1.33 (>=3 required)
  passes the checklist: false

Corrected round (after L2-L6):
  biased sample: false (dominant power_user (37.5%))
  average evidence strength: 3.67 (>=3 required)
  passes the checklist: true

The rushed round fails both checklist questions, not just one: the sample is dominated by power_user at 75% (exactly the result you already saw in lesson 4), and the collected evidence —two hypothetical pieces and one stated_opinion— has an average strength of just 1.33, well below the 3 threshold the checklist demands. Any conclusion drawn from this round would be built on a doubly weak foundation: people who were already predisposed to say yes, saying things that aren't even real behavior.

The corrected round passes both questions: the sample is balanced across three segments, with none above the dominance threshold, and the evidence —two observed_behavior pieces and one reported_behavior— reaches an average strength of 3.67. Notice the corrected round deliberately includes unfavorable evidence (the user who ignored the section entirely) — the checklist doesn't require all the evidence to be positive, it requires it to be representative and of the right type. A "corrected" round that only changed the sample but kept purely favorable evidence would still be suspect, even if it passed the first question.

The complete checklist, in two questions

1. Is the sample biased? — sampleBias(), lesson 4. If any segment dominates with 70% or more, or if a relevant segment is missing, the collected evidence —no matter how strong individually— represents only part of the real audience.
2. Is the evidence strong enough? — rankByStrength(), lesson 6. If most pieces are hypothetical or stated_opinion, even a perfectly representative sample is predicting future behavior from opinions, not actions.

The two questions are independent, and both have to pass. A representative sample with weak evidence, or strong evidence from a biased sample, produce the same final result: a conclusion that doesn't deserve the trust being placed in it.

Common mistakes

Applying the checklist after the decision's already been made, to justify it. What happens: the team has already informally decided recommendations "looks good" — and runs honestyChecklist() not to decide whether to trust the evidence, but to have a document backing up a decision that was already made, picking which evidence to include in the analysis based on the result they want to get. Why it happens: a checklist with green checkmarks feels like objective validation, and it's tempting to build the checklist's input —which evidence goes in, how the sample gets defined— to produce the desired result, instead of letting the checklist honestly evaluate what was already collected. How to spot it: if the order of events was "we decided, then we ran the checklist" instead of "we ran the checklist, then we decided," the checklist is being used as theater, not as a tool. How to fix it: run honestyChecklist() over all the collected evidence and all the collected sample, before anyone on the team expresses an opinion about which conclusion they prefer — the order matters as much as the result.

Confusing "it passed the checklist" with "the hypothesis is validated". What happens: today's example's corrected round passes the checklist (passes: true), and someone concludes that means recommendations is already validated and ready to build. Why it happens: a checklist with a clear verdict (true/false) feels like a final answer, when in reality it only certifies the evidence is reliable enough to be synthesized — it doesn't yet say what conclusion that synthesis produces. How to spot it: ask what percentage of the behavior evidence was favorable versus unfavorable — in today's example, of the corrected round's three pieces, one of the two behavior observations was negative (ignored the section entirely); the checklist passes, but the conclusion about recommendations still isn't decided. How to fix it: honestyChecklist() certifies the input evidence's quality, not the output decision — counting signals, deciding whether they're enough, and updating confidence is explicitly module 7's job, which starts right where this checklist ends.

Exercises

Exercise 1 — Run the checklist over a mixed sample. With this round, manually calculate whether it would pass honestyChecklist():

const mixedRound = {
  sample: { segments: [
    { segment: 'power_user', count: 4 },
    { segment: 'new_user', count: 4 },
  ]},
  evidence: [
    { claim: 'Clicked the fake door', type: 'observed_behavior' },
    { claim: 'Said they would like to see it', type: 'hypothetical' },
  ],
};

See solution

Sample: total 8, power_user 50%, new_user 50% — no segment dominates (none ≥70%), none is at zero. sampleBias.biased = false, the first question passes. Evidence: observed_behavior (strength 4) and hypothetical (strength 1), average = (4+1)/2 = 2.5, below the 3 threshold. strongEnough = false, the second question fails. passes: false — even with a perfectly balanced sample, the collected evidence is, on average, too weak to pass the full checklist. This case shows exactly lesson 6's exercise 3's point: a healthy sample doesn't guarantee the evidence is strong enough.

Exercise 2 — Design a round that fails only because of the sample. Design a round object (with sample and evidence) where rankByStrength() would give an average strength of at least 3, but sampleBias() would flag biased: true. No need to run Node — manually verify both criteria separately.

See solution

A valid solution: sample: { segments: [{segment: 'power_user', count: 9}, {segment: 'new_user', count: 1}] } (90% power_user, clearly above the 70% dominance threshold → biased: true) together with evidence: [{claim: 'Clicked', type: 'observed_behavior'}, {claim: 'Added to cart', type: 'observed_behavior'}] (both strength: 4, average = 4, above the 3 threshold → strongEnough: true). The result: passes: false, because even though the evidence is of the strongest possible type, it comes almost exclusively from power_user — exactly the case that shows why the checklist's two questions are independent and both necessary.

Exercise 3 — Explain why the order of the questions doesn't matter. honestyChecklist() evaluates sampleBias() and rankByStrength() in the same step, with neither depending on the other's result. Would the final verdict (passes) change if evidence strength were evaluated first and the sample afterward, instead of the other way around?

See solution

It wouldn't change. passes is defined as !sample.biased && strongEnough — a logical AND operation between two independent conditions. JavaScript's && operator evaluates left to right, but an AND expression's final result is the same no matter the order of its operands: it's true only if both conditions are true, and false if either one is false. This reflects an important conceptual point of this lesson: the checklist's two questions —is the sample okay? is the evidence strong?— are checks independent of each other, not a sequence where one determines the other.

Summary and next step

This lesson added no new model: it combined sampleBias() (lesson 4) and rankByStrength() (lesson 6), unchanged, into a two-independent-question checklist. You saw the full contrast between recommendations's rushed round —which fails both questions, with a sample dominated 75% by power users and an average evidence strength of just 1.33— and the corrected round, which passes both, with a balanced sample and evidence mostly of observed behavior, honestly including an unfavorable signal.

Before moving on you should be able to: recite the checklist's two questions from memory; and explain why "it passed the checklist" certifies the input evidence's quality, not the final conclusion about the hypothesis.

With this, you close the module's six content lessons. Lesson 8, the mini-project, asks you to audit recommendations's full discovery plan with these same two tools —sampleBias() and rankByStrength(), with no changes at all— and fix it, leaving the evidence ready for module 7 to synthesize and finally update the confidence that's stayed capped at 0.3 since this guide's first module.

Resources

• Paul Saffo, interview "Betting on Strong Opinions, Weakly Held" — saffo.com/interviews. On the discipline of subjecting any conclusion of your own to an honest review before trusting it — the spirit behind a checklist run before deciding, not after. In English.
• Teresa Torres, "Assumption Testing: Everything You Need to Know to Get Started" — producttalk.org/assumption-testing. On why testing assumptions in a disciplined way, with explicit criteria, is among the highest-value practices a product team can adopt. In English.
• Nielsen Norman Group, "Confirmation Bias in UX" — nngroup.com/articles/confirmation-bias-ux. Revisit it here as a close: its practical recommendations —research instead of validate, keep an open mind— are, in essence, what this two-question checklist turns into a verifiable check. In English.