Decision Matrix template

Overview

Across M7 you saw 5 specific trade-offs (scale up/out, cache/compute, managed/self-hosted, serverless/containers, model routing). Each with its own framework. Now we unify them into one tool: the Decision Matrix.

A Decision Matrix is a structured document where you list options, define weighted criteria, score each option, calculate a composite score, and document the decision. It's what a Tech Lead or Architect presents when asked "why did you choose X and not Y?".

This lesson builds the specific template you'll use in M8 (Capstone) and in any future project.

By the end you'll be able to:

• Build a Decision Matrix template for any technical decision
• Assign weights based on context (early-stage vs enterprise vs solo dev)
• Score options with quantifiable criteria
• Document accepted decisions and accepted trade-offs

The anatomy of a Decision Matrix

# Decision: [Decision title]

## Context
- Why we need to decide this
- Applicable constraints (budget, time, compliance)

## Options
1. **Option A**: brief description
2. **Option B**: brief description
3. **Option C**: brief description

## Criteria
- **Cost** (weight: 0.X)
- **Performance** (weight: 0.X)
- **Complexity** (weight: 0.X)
- ...

## Evaluation matrix

| Criterion | Weight | Option A | Option B | Option C |
|----------|------|----------|----------|----------|
| Cost | 0.30 | 4 ($Y) | 3 ($Z) | 2 ($W) |
| Performance | 0.25 | 5 (Xms) | 4 (Yms) | 3 (Zms) |
| Complexity | 0.20 | 3 | 4 | 5 |
| Vendor lock-in | 0.15 | 5 | 4 | 3 |
| Team familiarity | 0.10 | 4 | 5 | 2 |
| **Weighted score** | | 4.20 | 3.80 | 3.00 |

## Decision
**Selected: Option A**

## Trade-offs accepted
- We accept a higher upfront cost ($Y) in exchange for better performance and less lock-in
- We accept slightly higher complexity to have downstream flexibility

## Reconsider if
- Volume grows >5×
- Vendor changes pricing significantly
- A new option (D) appears in the market

Defining criteria: the most important part

Don't use generic criteria. "Simplicity" is ambiguous. Use measurable and specific criteria.

Bad example (vague)

Impossible to evaluate later. How "low" is low? "Good" compared to what?

Good example (quantifiable)

Now you can evaluate objectively.

Scoring system: 1-5 with anchors

Each option is scored 1-5 per criterion. Define explicit anchors:

Cost (1-5):
  5 = excellent, <$500/month
  4 = good, $500-1000/month
  3 = acceptable, $1000-2500/month
  2 = expensive, $2500-5000/month
  1 = very expensive, >$5000/month

With anchors, you and other team members score the same. Without anchors, "high/low" is subjective.

Weights: what changes based on context

The weights reflect what matters to your organization now. Three typical examples:

Early-stage startup

Speed to market: 0.30
Cost: 0.25
Team familiarity: 0.20
Performance: 0.15
Vendor lock-in: 0.10

Lock-in barely matters (you're validating the product). Speed to market critical.

Mid-stage scaling

Cost: 0.30
Performance: 0.25
Vendor lock-in: 0.20
Team familiarity: 0.15
Speed to market: 0.10

Cost matters more (you have volume). Lock-in critical (switching is expensive).

Enterprise / regulated

Compliance: 0.30
Performance: 0.25
Cost: 0.15
Vendor lock-in: 0.15
Operational: 0.15

Compliance dominates. Cost is just one factor.

Rule: weights must sum to 1.0. Without this, your score isn't interpretable.

Composite score calculation

Score(option) = Σ (criterion_weight × criterion_score)

Example with real numbers:

Score Pinecone = 0.30×3 + 0.15×5 + 0.20×5 + 0.20×4 + 0.15×2 = 0.9 + 0.75 + 1.0 + 0.8 + 0.3 = 3.75

Score ChromaDB = 0.30×4 + 0.15×3 + 0.20×2 + 0.20×4 + 0.15×5 = 1.2 + 0.45 + 0.4 + 0.8 + 0.75 = 3.6

The difference is small (3.75 vs 3.6), which indicates it's a close decision. Either option is defensible. What decides the final choice may be a qualitative factor not in the matrix.

Documenting accepted trade-offs

After choosing, write explicitly what you're sacrificing:

## Trade-offs accepted

By choosing Pinecone:
- ✅ Speed to market: 1-hour setup vs 8 hours
- ✅ Zero operational overhead
- ⚠️ Vendor lock-in: 6-week migration if needed
- ⚠️ Higher recurring cost: $800/mo vs ~$600/mo TCO of self-hosted

We accept the lock-in risk because:
- We're early stage, value speed
- Pinecone has stable history (4+ years, $1B+ funding)
- Migration to ChromaDB later would take ~4 weeks, manageable cost

This is what differentiates a professional decision from a whim. You demonstrate you thought about alternatives and consciously accepted the consequences.

Reconsider triggers

When to revisit the decision:

## Reconsider this decision if:

1. **Volume**: vectors > 50M (Pinecone cost becomes $5K+/mo)
2. **Vendor**: Pinecone changes pricing model unfavorably
3. **Compliance**: client demands on-prem deployment
4. **Team**: hire DevOps lead, capacity for self-hosting
5. **Timeline**: Pinecone roadmap drops critical features

Without reconsider triggers, decisions become "always the same" out of inertia. With triggers, you know when to rethink.

Complete template (copy-paste)

# Decision: [Title]

## Status
- Proposed | Accepted | Rejected | Superseded
- Date: YYYY-MM-DD
- Decider(s): [Names]

## Context
[2-3 paragraphs: why we're deciding this, what problem is being solved]

## Constraints
- Budget: $X/mo
- Timeline: Y weeks to deliver
- Compliance: [requirements]
- Team: [skills available]

## Options considered

### Option A: [Name]
[1 paragraph of description]

### Option B: [Name]
[1 paragraph of description]

### Option C: [Name]
[1 paragraph of description]

## Evaluation criteria

| Criterion | Weight | Anchor (score 1-5) |
|-----------|--------|---------------------|
| Criterion 1 | 0.X | 1=worst, 5=best with definitions |
| Criterion 2 | 0.X | ... |
| Criterion 3 | 0.X | ... |
| Criterion 4 | 0.X | ... |
| Criterion 5 | 0.X | ... |
| **Total** | **1.0** | |

## Evaluation matrix

| Criterion | Weight | Option A | Option B | Option C |
|-----------|--------|----------|----------|----------|
| Criterion 1 | 0.X | X (data) | X (data) | X (data) |
| Criterion 2 | 0.X | X | X | X |
| ... |
| **Weighted score** | | X.XX | X.XX | X.XX |

## Decision

**Selected: Option [X]**

## Rationale
[2-3 paragraphs: why this score wins, what makes it particularly appropriate for our context]

## Trade-offs accepted
- ✅ [What we gain]
- ⚠️ [What we sacrifice]
- ⚠️ [Another trade-off]

## Reconsider if
1. [Condition 1]
2. [Condition 2]
3. [Condition 3]

## References
- [Link to relevant data, benchmarks, pricing pages]
- [Links to similar decisions made in industry]

Multiple decisions: the complete system

To design the Capstone (M8), you'll make many decisions:

Each one deserves its own Decision Matrix (or an ADR if it's more qualitative). In M8 you'll collect them all as an appendix of the Capstone Architecture Design.

Common traps

Trap 1 — A matrix without quantifiable numbers. "Good/Bad" instead of "$1200/mo, P95 200ms". Without numbers, the score is opinion in disguise.

Trap 2 — Unjustified weights. "Cost weight = 0.30 just because". Justify it: "Cost = 0.30 because we're pre-revenue and any $1K/mo worries us".

Trap 3 — An artificial score to favor your preferred option. Confirmation bias: you score the option you already chose generously. Do the exercise with honesty, trying to be fair.

Trap 4 — Ignoring trade-offs after deciding. "We chose X" without documenting what you sacrifice. Later, in production, you discover the painful trade-off. Document it explicitly.

Trap 5 — Reconsider triggers that are never evaluated. The triggers exist but nobody checks them. Set a calendar reminder every 6 months to review.

Trap 6 — A Decision Matrix as a post-hoc justification. You decided beforehand and put together the matrix to "prove" your decision is correct. Bad. Build the matrix before deciding.

Exercise

Design the Decision Matrix for your Capstone Architecture (M8). Decide:

Topic: which LLM provider for your AI Knowledge Assistant?

Context: an AI assistant for 50 enterprise clients, 1M queries/month expected.

Options:

• A: OpenAI (gpt-4o-mini default, gpt-4o for complex)
• B: Anthropic (Claude Haiku + Sonnet)
• C: Self-hosted (Mistral on Modal with A100)

Fill in the template:

1. Criteria and weights
2. Scores per option (with numbers, not generic)
3. Decision with rationale
4. Accepted trade-offs
5. Reconsider triggers

# Decision: LLM Provider for AI Knowledge Assistant

## Status: Accepted | Date: 2026-05-11

## Context
Need to choose primary LLM provider for production. Expected volume: 1M
queries/mo across 50 enterprise clients. Mix of simple lookups (60%) and
complex queries (40%).

## Constraints
- Budget: ~$3000/mo for LLM
- Compliance: SOC2 required, no data retention for training
- Multi-model strategy possible
- Latency P95 target: <8s

## Options

### Option A: OpenAI (gpt-4o-mini default, gpt-4o complex)
Hybrid routing: ~60% gpt-4o-mini, ~40% gpt-4o based on query type.

### Option B: Anthropic (Haiku + Sonnet)
Similar hybrid with Claude models.

### Option C: Self-hosted Mistral on Modal A100
Llama 8B for simple, fallback to managed for complex.

## Evaluation criteria

| Criterion | Weight | Anchor |
|-----------|--------|--------|
| Cost monthly | 0.25 | 5=<$1K, 4=$1-2K, 3=$2-3K, 2=$3-5K, 1=>$5K |
| Quality | 0.25 | 5=>90% pass rate, 4=85-90%, 3=80-85%, 2=70-80%, 1=<70% |
| Latency P95 | 0.15 | 5=<3s, 4=3-5s, 3=5-8s, 2=8-12s, 1=>12s |
| Vendor lock-in | 0.15 | 5=easy switch (1wk), 1=very hard (8wk+) |
| Compliance | 0.10 | 5=full SOC2+HIPAA+EU, 1=nothing |
| Operational | 0.10 | 5=zero ops, 1=heavy ops |

## Evaluation matrix

| Criterion | Weight | OpenAI | Anthropic | Self-hosted |
|-----------|--------|--------|-----------|-------------|
| Cost | 0.25 | 4 ($1.8K) | 3 ($2.5K) | 3 ($2.5K incl. infra) |
| Quality | 0.25 | 5 (eval: 88%) | 5 (eval: 90%) | 3 (eval: 78%) |
| Latency P95 | 0.15 | 4 (3.5s) | 4 (4.2s) | 3 (6s incl. cold start) |
| Vendor lock-in | 0.15 | 2 (medium-hard) | 2 | 5 (open source, easy) |
| Compliance | 0.10 | 5 | 5 | 4 (depends on Modal) |
| Operational | 0.10 | 5 | 5 | 3 (some ops on Modal) |
| **Weighted** | | **4.15** | **3.9** | **3.40** |

## Decision: **OpenAI** (Option A)

## Rationale
OpenAI wins by a small margin (4.15 vs 3.9 Anthropic). The decisive factor:
- The team's familiarity with OpenAI APIs
- Better cost positioning (gpt-4o-mini is the cheapest of the managed quality options)
- Eval set quality is competitive with Anthropic

Anthropic is the obvious backup if OpenAI has problems. Self-hosted discarded:
the quality gap is too large for B2B enterprise.

## Trade-offs accepted
- ✅ Speed to market: zero infra to operate
- ✅ Consistent quality
- ⚠️ Vendor lock-in: 4-6 weeks to migrate to Anthropic if needed
- ⚠️ Cost growth: $1.8K with 1M queries; at 5M queries probably $7K+/mo
- ⚠️ Dependence on OpenAI uptime and pricing decisions

## Reconsider if
1. Volume exceeds 5M queries/mo (cost crossover with self-hosted possible)
2. OpenAI changes terms or pricing significantly
3. Client demands on-prem (would need Mistral self-hosted EU)
4. New Mistral/Llama models match quality with a major cost advantage
5. Internal team grows with DevOps capacity for self-hosting

## References
- Eval set results: [link]
- OpenAI pricing: https://openai.com/pricing
- Anthropic pricing: https://anthropic.com/pricing
- Modal pricing: https://modal.com/pricing

Summary

You learned:

• ✅ The anatomy of a Decision Matrix: context, options, criteria, scores, decision, trade-offs, reconsider
• ✅ Quantifiable vs vague criteria
• ✅ A scoring system with anchors (1-5 with definitions)
• ✅ Weights based on context (early-stage vs scaling vs enterprise)
• ✅ Documenting trade-offs explicitly
• ✅ Reconsider triggers to revisit decisions
• ✅ A complete copy-paste template
• ✅ Traps: a vague matrix, unjustified weights, confirmation bias

Checkpoint: if you can generate a complete Decision Matrix for any decision in your stack, you're ready.

Next lesson

08 — Project: Trade-off Decision Matrix. You apply the whole module to a concrete case: an AI system with multiple technical decisions. You produce the final document that becomes part of M8 (Capstone Architecture Design).

Resources

1. Architectural Decision Records template.
2. TOPSIS method — an alternative scoring method.
3. Multi-criteria decision analysis.
4. How to write an ADR.
5. Decision matrix examples.