Module 5: Stakeholders and Quality Attributes

3. The quality attribute scenario: making the vague measurable

Next →

Overview

By the end of this lesson you'll know how to close the gap between a prioritized attribute and something a team can actually build and verify —because there is a gap, and it's where half of quality requirements die—. In lesson 2 you came out with a ranking: scalability first, then security, availability, performance, cost. But "high scalability" isn't a requirement. It's an adjective stuck to an attribute. It doesn't tell the team how much to scale, in response to what stimulus, under what conditions, or how we'll know if we achieved it. It's as vague as "I want a comfortable house": comfortable for whom, in what climate, with what budget? The tool that closes that gap is called the quality attribute scenario and it comes from Bass, Clements, and Kazman. It's a six-part template that turns a vague attribute into a concrete and testable sentence: source (who or what generates the stimulus), stimulus (what happens), artifact (what part of the system receives the stimulus), environment (under what conditions), response (what the system must do), and —the most important— response measure (with what number is it measured that it did so). With the six parts, "high availability" becomes: "when the external payment gateway (source) stops responding (stimulus), the checkout service (artifact), during the Black Friday peak (environment), retries and queues the charge without losing the order (response), completing 99.95% of the orders with less than 5 seconds of extra latency (response measure)". That can be designed. That can be tested.

This matters because the sixth part —the response measure— is the line that separates a requirement from a wish, and almost no one respects it. An attribute with no measure can't be designed (how much redundancy is needed for "high availability"? no one knows, because "high" isn't a number), can't be tested (did it pass the availability test? no one can say, because there's no threshold), and can't be defended (did we meet what the business asked for? impossible to know). An attribute with a measure becomes a contract: "99.95% of the orders, under 5 seconds" is something you design to meet, test against a number, and defend with evidence. This lesson's rule is hard and holds for the whole craft: if a quality requirement has no measure, it's not a requirement —it's a disguised wish—, and treating it as a requirement is building on sand. You'll execute a validation that takes three Mercado scenarios and separates the ones ready to design from the ones still wishes for lack of that measure.

Connection with the module: this lesson takes the ranking from lesson 2 and makes it usable. Prioritizing the attributes (lesson 2) answers what to optimize and in what order; making each attribute a measurable scenario answers how much and how we'll verify it. Without this step, the ranking is a list of good intentions no team can execute. With this step, each prioritized attribute becomes one or several concrete scenarios the team designs and tests. The lessons that follow lean on this: lesson 4 asks which of these scenarios are architecturally-significant (shape the structure) and which aren't; lesson 5 discovers implicit scenarios no one wrote; and lesson 6 uses the response measure as the number that's translated to the business's language ("99.95% availability" becomes "X dollars of protected sales"). The frontier stays: here you learn to write the scenario that makes an attribute measurable; how the exact level of the measure is decided when it competes with another attribute —99.95% against 99.99%— is architecture-decisions's method.

The contractor who needs a number, not an adjective

Think about it with whoever builds the house, not with whoever designs it. The architect hands the blueprints to the contractor —the one who's actually going to raise the walls— with a note that says: "the house must be cool in summer". The contractor reads it and can't do anything with it. Cool how? With air conditioning, roof insulation, north-facing windows, thick walls? Cool in a Monterrey summer at 42 degrees or a Bogotá one at 22? How much is "cool" —24 degrees inside when it's 40 outside, or is 30 enough? The adjective "cool" is a legitimate wish, but the contractor doesn't build wishes: they build specifications. So they return the note with a question: "give me a number". And the architect rewrites: "when the outside temperature reaches 40 °C (environment), in the second-floor rooms (artifact), the house's passive system (response: insulation + cross-ventilation, no air conditioning) must keep the inside below 26 °C (response measure)". Now the contractor knows what to build, and —key— when they finish, someone can measure with a thermometer whether the house complies. "Cool" can't be measured with a thermometer; "below 26 °C with 40 outside" can.

That's exactly the difference between a quality attribute and a quality attribute scenario. "High availability", "good performance", "secure" are the adjectives the business and the architect use to talk —they're useful for prioritizing (lesson 2)—, but no team builds an adjective. The team needs the number: not "available" but "99.95% of orders completed"; not "fast" but "checkout responds in under 2 seconds for the 95th percentile"; not "secure" but "100% of accesses to another vendor's data are blocked and logged". The six-part scenario is the architect's rewritten note: it takes the business's adjective and adds the five things missing to make it buildable and —above all— verifiable with a thermometer. Without the number at the end, you're asking the team to build "cool in summer", and everyone will understand something different.

Worked example: validating that a scenario is measurable

We'll take three Mercado scenarios, each with its six parts (or almost), and execute a validation that verifies two things: that they're complete (have the six parts) and that they're measurable (have a real response measure, not empty). The third scenario has a deliberate trap —it's missing the measure— so you see how the validator catches it. The structure is a dictionary per scenario; the validation counts the parts present and checks whether there's a measure.

# A vague attribute ("we want high availability") can't be tested or
# designed. The 6-part SCENARIO (Bass/Clements/Kazman) makes it concrete and
# measurable. We validate which scenarios are complete and have a measure.

# Each scenario: source, stimulus, artifact, environment, response, response_measure.
scenarios = [
    {
        "attribute": "availability",
        "source": "the external payment gateway",
        "stimulus": "stops responding",
        "artifact": "the checkout service",
        "environment": "Black Friday peak",
        "response": "retries and queues the charge without losing the order",
        "response_measure": "99.95% of orders completed, < 5 s extra latency",
    },
    {
        "attribute": "security",
        "source": "an authenticated external vendor",
        "stimulus": "requests ANOTHER vendor's data via API",
        "artifact": "the multi-tenant API gateway",
        "environment": "normal operation",
        "response": "rejects and logs the attempt",
        "response_measure": "100% of cross-tenant accesses blocked and audited",
    },
    {
        "attribute": "scalability",
        "source": "the marketing team",
        "stimulus": "launches a campaign that 10x's the traffic",
        "artifact": "the catalog",
        "environment": "production",
        "response": "the system scales without going down",
        "response_measure": "",   # <-- vague: no measure, can't be tested
    },
]

required = ["source", "stimulus", "artifact", "environment", "response", "response_measure"]

print(f"{'attribute':<13}{'parts':>8}{'measurable?':>13}   verdict")
ready = 0
for sc in scenarios:
    present = sum(1 for p in required if sc.get(p))
    measurable = bool(sc.get("response_measure"))
    ok = present == len(required) and measurable
    ready += ok
    verdict = "ready to design" if ok else "VAGUE: missing the measure"
    print(f"{sc['attribute']:<13}{present:>5}/6{('yes' if measurable else 'NO'):>13}   {verdict}")

print()
print(f"{ready}/{len(scenarios)} scenarios are ready to design and test.")
print("The one with no 'response_measure' isn't a requirement: it's a wish.")

What to expect. Running it:

attribute       parts  measurable?   verdict
availability      6/6          yes   ready to design
security          6/6          yes   ready to design
scalability       5/6           NO   VAGUE: missing the measure

2/3 scenarios are ready to design and test.
The one with no 'response_measure' isn't a requirement: it's a wish.

Stop at the third row, because it's the whole lesson. The scalability scenario has five of the six parts —source (marketing), stimulus (campaign that 10x's the traffic), artifact (catalog), environment (production), response (scales without going down)— and still the validator marks it as VAGUE. Why? Because it's missing the one part that makes it a requirement: the response measure. "The system scales without going down" sounds like a requirement, but it isn't: it doesn't say how much traffic ("10x the traffic" isn't an absolute number —10x of what base?—), it doesn't say what "without going down" means (zero errors? less than 1% of errors? latency under what threshold?), and therefore no one can test whether it was met. Imagine the marketing campaign running and the team wondering "did we pass?": without a number, the answer is a discussion of opinions, not a measurement. The validator caught exactly what the human eye lets pass: a scenario that looks complete because it has five written parts, but that's a wish because it's missing the sixth.

Compare with the availability scenario, which did pass. It has the six parts, and the sixth is a real number: "99.95% of orders completed, less than 5 seconds of extra latency". With that, the team knows what to build (enough redundancy and retries to not lose more than 0.05% of the orders), knows how to test it (simulate the gateway's outage in a peak and measure how many orders complete and with how much latency), and the business knows what it received (a quantified guarantee, not a vague promise). Same with the security one: "100% of cross-tenant accesses blocked and audited" is a number —100%— you can design with (strict isolation) and test (attempt a thousand cross accesses and verify all thousand were blocked and logged). The difference between the two that passed and the one that failed isn't the writing or the effort: it's that one ended in a number and the other in an adjective.

Notice something subtle the validator teaches about the craft. The three scenarios are well written —all three identify the source, the stimulus, the artifact, the environment—. The scalability one isn't careless; someone thought it through. And yet it's useless as a requirement. That tells you that the easy part of a scenario is everything but the measure, and the hard part —the one that really matters— is the measure. It's easy to write "the system scales without going down"; it's hard to commit to "supports 10,000 requests per second with less than 0.1% errors and p95 latency under 300 ms". The hard is hard for a good reason: the number forces you to decide —to commit to a concrete threshold you can meet or miss—. The adjective avoids the decision; the number forces it. That's why so many people stay at the adjective: it's comfortable. The validator doesn't let them: no number, VAGUE.

An honest nuance. This example's validator checks that the measure exists (that the field isn't empty), not that it's good. A scenario could put a response measure that's a number but a useless number —"scales reasonably fast"— and this validator would let it pass because the field isn't empty. The mechanical validation catches the most common error (missing measure) but doesn't judge the quality of the measure; the human judgment does. A good measure is specific (a percentile, a percentage, a time threshold), bounded to a condition (under what load, in what environment), and verifiable (there's a way to measure it). "Under 5 seconds for 99.95% of the orders during Black Friday" meets the three. "Fast" meets none. The validator is your first line of defense; your judgment is the second.

Deep dive: why six parts, and where the missing ones come from

It's worth understanding why it's six parts and not fewer, because each one covers a hole through which vagueness escapes.

Source and stimulus answer "in response to what?". An attribute isn't unconditional: a system isn't "available" in the abstract, it's available in response to certain failures. Saying "when the external gateway (source) stops responding (stimulus)" bounds the requirement to a concrete event. Without this, "high availability" would have to cover all imaginable failures —including a meteorite—, which is impossible and therefore useless. The source and the stimulus make the requirement finite: not "resists everything", but "resists this specific failure".

Artifact and environment answer "where and under what conditions?". The same stimulus matters differently depending on which part receives it and when. The gateway failing matters a lot in checkout during Black Friday (artifact + environment) and almost not at all in an internal report on a Tuesday at 3am. Bounding the artifact and the environment concentrates the requirement where it really hurts, and avoids the mistake of demanding the same level of quality everywhere —which is the recipe for over-cost (lesson 7)—. The environment, besides, is usually where the most expensive requirement hides: "in normal operation" is easy; "during the Black Friday peak with triple the traffic" is where the architecture is decided.

Response and response measure answer "what must happen, and how much?". The response describes the desired behavior ("retries and queues without losing the order"); the response measure quantifies it ("99.95%, under 5 seconds"). We already saw the measure is the part that makes or breaks the scenario. But notice the pair: the response without the measure is an adjective ("scales without going down"), and the measure without the response is a number with no behavior ("99.95%" of what?). You need both: the behavior gives meaning to the number, the number gives precision to the behavior.

Now, the practical question: where do the parts the stakeholder doesn't give come from? Because the VP never says "source: external gateway, environment: Black Friday". The VP says "we can't go down on Black Friday". The other five parts are completed by the architect, and that's where a good part of the craft is. The "Black Friday" (environment) the architect got from knowing the business (they know that day concentrates the revenue). The "external gateway that stops responding" (source + stimulus) they got from knowing the system (they know the gateway is a typical point of failure in peaks). The "99.95% with under 5 seconds" (measure) they negotiated with the business by translating the cost of each level (lesson 6). Writing a good scenario is an act of translation and of knowledge: the stakeholder contributes the attribute and the critical environment; the architect contributes the structure of the six parts and, above all, proposes the measure that's then agreed. The scenario is the artifact where the business's wish and the architect's knowledge meet and become a verifiable contract.

And a note on the frontier. This scenario says "99.95% availability". Why 99.95% and not 99.9% or 99.99%? That's a decision with a huge trade-off —each extra nine costs a lot more (lesson 6)— and deciding the exact level when availability competes with cost is architecture-decisions's method. Here your job is to structure the scenario so it has a measure; which measure to choose, when that choice pits two attributes against each other, is the decision you document in an ADR with its matrix. The scenario is the mold; the number that goes in the mold, when there's a conflict, is decided by the other guide's method. What this lesson gives you is the discipline to demand that there be a number, and the structure to put it in its place.

Common mistakes

Confusing a well-written scenario with a measurable one (of false completeness). What happens: the architect writes a scenario with five beautiful parts —source, stimulus, artifact, environment, response, all detailed— and since it looks complete, they deem it good, even though the response measure is "the system responds well". The team implements it, and at test time no one can say whether it passed. Why it happens: five written parts give a sense of rigor that deceives; the eye sees work and assumes completeness. How to spot it: cover the first five parts with your hand and read only the response measure; if it's not a number you can run a test against, the scenario is vague however pretty the rest is. How to fix it: treat the response measure as mandatory and as the first thing you review —this lesson's validator does it mechanically—; without a measure, the scenario goes back to sender.

Putting a measure that isn't verifiable (of the decorative number). What happens: the scenario says "response measure: the system scales reasonably", or "responds in an acceptable time", and since there are words in the field, it seems measurable. But "reasonably" and "acceptable" aren't numbers: there's no way to run a test that gives yes or no. Why it happens: the pressure to fill the field leads to putting something, and an adjective shaped like a measure is easier than a real threshold that forces commitment. How to spot it: ask yourself "what test would I run, and what value would make it pass or fail?"; if you can't name the test and the threshold, the measure is decorative. How to fix it: demand that every measure be a number with a unit and a condition —percentile, percentage, milliseconds, under what load—; "p95 under 300 ms with 10,000 req/s" is verifiable, "fast" isn't.

Demanding the same level in all environments (of the ignored environment). What happens: the architect writes scenarios without bounding the environment, so "99.99% availability" applies equally to checkout on Black Friday and the internal admin panel on a Sunday. They end up designing (and paying for) extreme availability in parts that don't need it. Why it happens: omitting the environment feels simpler ("make everything very available"), but it's the door to over-cost. How to spot it: if your scenarios don't distinguish environments or artifacts —if the demanded level is the same everywhere—, you're not bounding the requirement where it hurts. How to fix it: use the artifact and environment parts to concentrate each level of quality where the business needs it; checkout on Black Friday deserves 99.95%, the internal report deserves 99% and that's fine —that differentiation is exactly what avoids promising everything at the maximum (lesson 7)—.

Exercises

Exercise 1 — Complete the scenario. Mercado's VP says: "the product search has to be fast, or people leave". Turn it into a six-part quality attribute scenario. Reasonably invent the parts the VP didn't give (source, artifact, environment) and —above all— propose a verifiable response measure.

See solution

A reasonable scenario (the invented parts are based on knowing the business and the system):

  • attribute: performance
  • source: a buyer
  • stimulus: types a search term and requests results
  • artifact: the catalog search service
  • environment: peak hour of normal traffic (not Black Friday)
  • response: returns the relevant results, sorted
  • response measure: the 95th percentile of searches responds in under 400 ms; the 99th percentile, in under 800 ms

The important part is the response measure. "Fast" (what the VP said) can't be tested. "Under 400 ms for the p95" can: you can run a load test that fires thousands of searches and measure the 95th percentile of the response time, and there's a clear threshold that makes it pass or fail. Note three good-measure decisions: I used a percentile (not an average, which hides the slow cases —the p95 says "95% of people saw something this fast or faster"), I gave it a unit (milliseconds), and I bounded it to an environment (normal peak hour; on Black Friday the threshold could relax or require another scenario). I also separated p95 and p99 because "fast for almost everyone" and "never too slow" are different guarantees worth declaring separately.

Exercise 2 — Catch the vague one. A colleague hands you this scenario for review: "When many users arrive (source: users; stimulus: lots of traffic; artifact: the system; environment: production), the system must handle it well (response) in a scalable way (response measure: scalable)". Without running code, point out why the validator would mark it as VAGUE and which parts, besides the measure, are too imprecise to be useful.

See solution

The validator would mark it VAGUE because the response measure is "scalable" —a word, not a number—: there's no threshold you can test. "Scalable" is exactly the adjective the scenario was supposed to eliminate, reappeared in the measure field. There's no test that gives "yes, it's scalable" or "no, it isn't" against that text.

But the problem is deeper: almost all the parts are too imprecise, not just the measure.

  • source: "users" — how many? from where? "Many users" isn't a bounded source. A good source would say, for example, "a marketing campaign that multiplies the base traffic by 10".
  • stimulus: "lots of traffic" — how much is a lot? Without a base number ("from 1,000 to 10,000 requests per second"), there's no measurable stimulus.
  • artifact: "the system" — too broad. The whole system, or the catalog, or the checkout? Scaling the catalog (read, cacheable) is a different problem from scaling the checkout (write, transactional). "The system" hides that difference.
  • environment: "production" — acceptable but poor; it doesn't say whether it's an expected peak or an exceptional event.
  • response: "handle it well" — another adjective. Is "well" without errors? with latency under a certain threshold? "Well" doesn't describe a verifiable behavior.

The lesson: the scenario is full of vague adjectives ("many", "lots", "well", "scalable") disguised as parts. A good scenario replaces each adjective with a specification: "from 1,000 to 10,000 req/s (stimulus) on the catalog (artifact), keep the p95 under 300 ms and less than 0.1% errors (response + measure)". The validator catches the empty measure; your judgment catches the rest.

Exercise 3 — From the measure to the design and the test. Take the availability scenario that passed the validation: "when the external payment gateway stops responding, during Black Friday, the checkout retries and queues the charge without losing the order, completing 99.95% of the orders with less than 5 s of extra latency". Without going into detailed implementation, answer: (a) what does the "99.95% without losing the order" force you to design?; (b) how would you test that it's met?; (c) why would the same scenario without the response measure not let you do either (a) or (b)?

See solution

(a) What it forces you to design. The "99.95% without losing the order, even when the external gateway goes down" forces you into an architecture where the order doesn't depend on the charge being immediate: you need to decouple accepting-the-order from charging-the-order. Concretely, something like durably queuing the order as soon as the customer confirms (so it's not lost if the gateway fails), and retrying the charge asynchronously until the gateway comes back. The "99.95%" sets the bar: the design has to tolerate the gateway's outage long enough not to lose more than 5 of every 10,000 orders. That number decides how much durability and how many retries are needed.

(b) How you'd test it. By simulating the stimulus in a load environment: you generate traffic equivalent to a Black Friday peak, in the middle of the test you deliberately take down the payment gateway (or its simulation) for a while, and you measure two things: how many orders completed (must be ≥ 99.95%) and how much extra latency the retry/queueing added (must be < 5 s). It's a chaos/resilience test with clear pass criteria, precisely because you have numbers to compare against.

(c) Why without the measure you couldn't do either (a) or (b). Without the "99.95% with under 5 s", the scenario would say "the checkout doesn't lose orders when the gateway fails" —a wish—. For (a), you wouldn't know how much resilience to design: tolerate 1 second of outage or 1 hour? lose zero orders (very expensive, maybe impossible) or 0.05%? The number sets the design objective. For (b), you wouldn't know what test to run nor what would make it pass: without a threshold, the chaos test would give a result ("99.7% of the orders completed") no one could judge as success or failure. The measure is what turns the scenario into a design objective and a test criterion at the same time —it's the bridge between the business's wish and something a team builds and verifies—. That is, in one phrase, why the sixth part is the one that makes or breaks the scenario.

Summary and next step

In this lesson you learned to close the gap between a prioritized attribute and something buildable: the six-part quality attribute scenario (source, stimulus, artifact, environment, response, response measure) that turns a vague adjective into a testable sentence. With the contractor who needs a number and not an adjective you saw that "cool in summer" can't be built but "below 26 °C with 40 outside" can, and that the thermometer —the measure— is what separates a requirement from a wish. You validated it by executing: three Mercado scenarios, two ready to design and one caught as VAGUE for lacking the response measure, even though it had the other five parts well written. You understood that the hard part of a scenario is the measure (because the number forces commitment while the adjective avoids it), that the six parts each cover a hole through which vagueness escapes, and that the architect completes the parts the stakeholder doesn't give with their knowledge of the business and the system.

Before moving on you should be able to: take a vague business attribute and write a six-part scenario with a verifiable response measure; distinguish a real measure (number, unit, condition, test) from a decorative one (a disguised adjective); use artifact and environment to bound each level of quality where the business needs it; and apply the hard rule —if it has no measure, it's not a requirement—.

What follows is a different filter. You now know how to prioritize attributes (lesson 2) and turn them into measurable scenarios (this lesson). But not all the requirements that arrive —nor all the scenarios you could write— shape the architecture. Some are structural decisions everything hangs from (isolating each vendor's data); others are product details changed in an afternoon without touching the architecture (the buy button's color). In lesson 4 you'll learn to separate the signal from the noise with the concept of architecturally-significant requirement (ASR): the requirement that shapes the structure, is expensive to change later, and is high-risk. You'll execute a filter that takes a list of Mercado requirements and separates the few that really matter for the architecture from the many that matter to the product but not to the structure —so you spend your architect energy where it really counts—.

Resources

← Previous2. From the business goal to the quality attributeNext →4. Architecturally significant requirements: the signal and the noise