Actionable vs. vanity metrics

Overview

This whole guide rests on a distinction that seems obvious until you apply it to a real dashboard: not every metric helps you decide something. A vanity metric is a number that goes up, looks good in a presentation, and doesn't tell you what to do differently tomorrow. An actionable metric is a number that, when it changes, tells you something about your users' real behavior —and therefore suggests a concrete action—.

The trap isn't that vanity metrics are false. totalRegisteredUsers —the historical total of people who signed up for Mercado— is a real, verifiable data point that probably took real effort to get. The problem is something else: it doesn't change what you'd do next. If totalRegisteredUsers goes from 550,000 to 551,000 this week, what do you decide differently? Probably nothing. If checkoutConversionRate goes from 3.2% to 2.8%, you do know what to do: investigate what broke in checkout.

How this connects to the module. This lesson opens the module with the central distinction that lessons 3 through 7 will develop from different angles: lesson 3 goes deeper into why a rate is stronger than a total (one of the two signals you'll use today), lessons 4 and 5 give you two complete frameworks for choosing metrics with structure, lesson 6 organizes actionable metrics into a tree, and lesson 7 closes with the full criterion for what makes a metric good. Today you build the simplest tool of all —a two-question classifier— and run it over Mercado's first candidate metrics.

An analogy: the odometer and the speedometer

Every car has two numbers on the dashboard that sometimes get confused. The odometer —the total kilometers driven since the car left the factory— can only go up. It never goes down, no matter how badly you drive today. It's real data and it even has its use —for example, knowing when the next service is due—, but it tells you absolutely nothing about how you're driving right now. A car with 200,000 km could be going 40 km/h carefully, or could be about to crash at 160.

The speedometer, on the other hand, tells you the current speed: a number that goes up and down, that reflects what you're doing at this exact moment, and that you can act on immediately —brake, accelerate—. Nobody decides whether they're going too fast by looking at the odometer. totalRegisteredUsers is Mercado's odometer: it always grows, it's accumulated history, not action. checkoutConversionRate is the speedometer: it tells you how you're driving the business today, and gives you something to act on if you don't like the number.

Worked example: classifyMetric() over Mercado's first candidates

While putting together the recommendations measurement plan, the Mercado team gathers a first short list of candidate metrics. Before deciding which go on the experiment dashboard, let's build a simple classifier —a pocket heuristic, not a statistical algorithm— with two signals:

1. Is it a cumulative total that only ever goes up? If it can never go down because it sums from day 1, that's a strong signal of vanity.
2. Is it a rate (has a numerator and a denominator), comparable across periods, and tied to a real, recent user decision? If so, that's a strong signal that it's actionable.

When neither strong signal applies —it's not a cumulative total, but it's not a rate either— we use a fallback signal: if it still reflects a real, recent decision (for example, a count that resets every week, like how many distinct buyers bought this week), we treat it as actionable; if not, as vanity.

// classifyMetric: pedagogical heuristic with 2 strong signals + 1 fallback signal.
// It is NOT a statistical classifier or a universal rule -- it's a pocket rule
// for auditing candidate metrics before putting them on a dashboard.
function classifyMetric(m) {
  // Signal 1 (strong): a cumulative total that only goes up since day 1 -> vanity.
  if (m.cumulativeTotal) return 'vanity';
  // Signal 2 (strong): a rate/ratio, comparable across periods, tied to a real decision -> actionable.
  if (m.isRate && m.tiedToRecentBehavior) return 'actionable';
  // No strong signal: neither a cumulative total nor a rate. It's a count that resets each period.
  // It's saved ONLY if it's tied to a real, recent user decision.
  return m.tiedToRecentBehavior ? 'actionable' : 'vanity';
}

const candidates = [
  { name: 'totalRegisteredUsers', cumulativeTotal: true, isRate: false, tiedToRecentBehavior: false },
  { name: 'totalPageviewsAllTime', cumulativeTotal: true, isRate: false, tiedToRecentBehavior: false },
  { name: 'checkoutConversionRate', cumulativeTotal: false, isRate: true, tiedToRecentBehavior: true },
  { name: 'weeklyActiveBuyers', cumulativeTotal: false, isRate: false, tiedToRecentBehavior: true },
];

console.log('=== classifyMetric over 4 Mercado candidate metrics ===\n');
const results = candidates.map((m) => ({ ...m, label: classifyMetric(m) }));
results.forEach((r) => console.log('  ' + r.name.padEnd(24) + '-> ' + r.label));

const actionableCount = results.filter((r) => r.label === 'actionable').length;
const vanityCount = results.filter((r) => r.label === 'vanity').length;
console.log('\nTotal: ' + actionableCount + ' actionable, ' + vanityCount + ' vanity.');

What to expect. When you run the file with Node, the output is exactly this:

=== classifyMetric over 4 Mercado candidate metrics ===

  totalRegisteredUsers    -> vanity
  totalPageviewsAllTime   -> vanity
  checkoutConversionRate  -> actionable
  weeklyActiveBuyers      -> actionable

Total: 2 actionable, 2 vanity.

Notice the weeklyActiveBuyers case: it's not a rate —it's a count, 4100 distinct buyers, say—, so it doesn't trigger signal 2. But it's not a total that only goes up either: every week it starts over at zero and gets recalculated only with what happened that week. And since it's also directly tied to a real decision (buying), the fallback signal classifies it as actionable. Compare it with totalRegisteredUsers: that number could, in theory, also be counted "per week" —how many signed up this week—, but as defined in the candidate above it's a historical accumulation, so it falls straight into vanity through signal 1, without even reaching the second one.

Why vanity metrics are dangerous, not just useless

You might think: "well, if totalRegisteredUsers doesn't help decide anything, I just won't look at it, and that's that." The problem runs deeper than that. A vanity metric isn't neutral: it tends to go up on its own, almost regardless of what the team does well or poorly, because it's an accumulation that rarely goes down. That makes it dangerously easy to present as success. A team can have a bad quarter —conversion dropping, retention dropping, the complaints closet growing— and still close the presentation with "we passed half a million registered users!", a true sentence that hides everything else.

That's the real reason this module starts here, before any framework or any metric tree: if you don't first learn to spot a vanity metric, no framework will save you from choosing badly, because HEART and AARRR (lessons 4 and 5) also have categories where it's easy to slip in a total disguised as progress.

Common mistakes

Celebrating a cumulative total that only goes up as if it were a health signal. What happens: a product report opens with "we hit 550,000 registered users" as the headline of success, without mentioning any recent behavior rate. Why it happens: a big, growing number is easy to communicate and always "looks good", even in a bad quarter. How to spot it: ask yourself whether that number could have gone up all the same even if the product had gotten worse this week —if the answer is yes, it's vanity. How to fix it: require every report headline to come with at least one recent, comparable rate, as classifyMetric did with checkoutConversionRate against totalRegisteredUsers.

Assuming any number that "sounds like behavior" is already actionable. What happens: someone defends totalPageviewsAllTime by saying "but that's people using the product, right?", treating accumulated past usage as if it were comparable to recent usage. Why it happens: "pageviews" sounds like real activity, and it's easy to forget that without bounding it to a period, it's still a total that only grows. How to spot it: ask "can this metric go down next week if the product gets worse?" If the answer is no —because it's a historical accumulation—, it doesn't matter how "active" it sounds, it's still vanity. How to fix it: always bound it to a defined period (pageviews this week, not pageviews since launch) before evaluating whether it's also tied to a real decision.

Dismissing a count (not a rate) outright just because it isn't a percentage. What happens: in the opposite direction, someone sees weeklyActiveBuyers —a whole number, not a percentage— and writes it off as vanity just because "good metrics are rates". Why it happens: lesson 3 (which comes next) is going to insist heavily on rates, and it's easy to over-apply that rule ahead of time. How to spot it: the objection is "this isn't a rate" without asking whether it still resets every period and reflects a real decision. How to fix it: as you saw in today's example, a periodic (non-cumulative) count tied to real behavior —like weeklyActiveBuyers— can be actionable, even though an equivalent rate is almost always even stronger. That's exactly what lesson 3 explains next.

Exercises

Exercise 1 — Classify by hand. For each metric, decide whether classifyMetric() would mark it actionable or vanity, and explain which signal (1, 2, or the fallback) drives it:

• (a) totalOrdersAllTime — the historical count of every order placed on Mercado since launch, not bounded to any period.
• (b) monthlyReturnRate — the percentage of purchases that end in a return, calculated every month.
• (c) newSellersThisMonth — how many new sellers joined this month (resets every month, not a percentage).

See solution

• (a) vanity, by signal 1. It's a historical cumulative total (cumulativeTotal: true) that can only go up; signal 2 never even gets evaluated.
• (b) actionable, by signal 2. It's a rate (isRate: true) with a numerator (returns) and a denominator (purchases), comparable month over month, and directly tied to a real decision by buyers.
• (c) actionable, by the fallback signal. It's not a cumulative total (it resets every month) and it's not a rate either, but it's tied to a real decision (a seller decided to join) and it's comparable period over period, so the fallback signal classifies it as actionable.

Exercise 2 — Find the disguise. A teammate proposes measuring recommendations's success with totalRecommendationImpressionsAllTime —how many times, in total, since the carousel launched, a recommendation has been shown to any user—. Explain, using this lesson's vocabulary, why this metric is a vanity metric in disguise even though it "sounds" related to the feature.

See solution

totalRecommendationImpressionsAllTime is a cumulative total since launch: it can only go up, every time the carousel is shown to any user, regardless of whether that user clicked, bought, or closed the tab immediately. It exactly satisfies signal 1 of classifyMetric (cumulativeTotal: true) and is therefore classified vanity without even looking at anything else. The disguise is that "recommendation impressions" sounds directly related to the feature being measured —unlike something obviously unrelated like totalPageviews—, but being related to the feature doesn't make it actionable: it still doesn't say whether the carousel is working this week. The actionable equivalent would be something like weeklyRecommendationClickRate (clicks ÷ impressions, per week): that one has a numerator, a denominator, and a comparable time window.

Exercise 3 — Design the counterexample. Write a metric object (with the three fields cumulativeTotal, isRate, tiedToRecentBehavior) that classifyMetric() would classify as vanity despite being tied to a real, recent user decision (tiedToRecentBehavior: true). What does that tell you about which of the three signals carries the most weight in the heuristic?

See solution

Any metric with { cumulativeTotal: true, isRate: false, tiedToRecentBehavior: true } works as a counterexample —for example, totalPurchasesAllTime, the total historical count of purchases made on Mercado since launch—. Even though each individual purchase is indeed a real, recent user decision at the moment it happens, classifyMetric() never even gets to look at tiedToRecentBehavior because cumulativeTotal is true and the function returns at the first if. This shows that signal 1 (cumulativeTotal) has absolute priority in the heuristic: no amount of "this is tied to real behavior" saves a metric if it's expressed as a historical accumulation with no time window. The practical lesson: it's not enough for the underlying event to be real and actionable —the purchase itself is—; it's the way you aggregate that event into a metric (endless accumulation vs. a rate or a per-period count) that decides whether the resulting metric is useful for deciding anything.

Summary and next step

In this lesson you defined the module's central distinction: a vanity metric is a total that only goes up and doesn't tell you what to do differently; an actionable metric is a signal —ideally a rate, and failing that a periodic count tied to real behavior— that changes with what people do and suggests an action. You built and ran classifyMetric(), a pocket heuristic with two strong signals and a fallback, and ran it over Mercado's first four candidate metrics: two vanity (totalRegisteredUsers, totalPageviewsAllTime), two actionable (checkoutConversionRate, weeklyActiveBuyers).

Before moving on you should be able to: explain in your own words the difference between a cumulative total and a comparable rate; apply classifyMetric() by hand to a new metric; and recognize when a metric "sounds" related to a feature without actually being actionable.

Lesson 3 takes today's signal 2 —"is it a rate?"— and develops it in depth: why an absolute total, even one that isn't an infinite historical accumulation, is still weaker than a rate with its denominator visible.

Resources

• Alistair Croll and Benjamin Yoskovitz, Lean Analytics — leananalyticsbook.com/tag/vanity-metrics. The official site for the book that popularized the term "vanity metric" and the distinction with actionable metrics this lesson develops. In English.
• Eric Ries, "Vanity Metrics vs. Actionable Metrics" (the original argument from The Lean Startup, summarized) — cited and expanded in the same Croll and Yoskovitz book above; the idea that "if a metric can't change your behavior, it's a bad metric" originates there.
• ProductPlan, "Vanity Metrics" — productplan.com/glossary/vanity-metrics. A short glossary with additional examples by industry, useful for practicing classification with metrics outside Mercado. In English.