Advanced decision matrix

You have three measured dimensions: latency, cost, quality. Each provider wins on one and loses on another. The question remains: which do you choose?

The answer isn't to pick the "best at everything" (it doesn't exist). It's to decide how you weight the dimensions and apply that weighting with discipline. This capsule gives you the method.

By the end you'll be able to:

• Assign weights to the dimensions according to your project's real priorities
• Normalize the metrics so they're comparable (latency in seconds doesn't add up with cost in dollars)
• Calculate a composite score that ranks providers
• Document the decision with reasons that survive a technical audit
• Distinguish absolute constraints (privacy, compliance) from preferences (latency, cost)

Why it matters

When a team picks a provider "by consensus", the outcome depends on who spoke loudest in the meeting. When it picks with a weighted, documented matrix, the outcome is defensible:

• Three months later, someone asks "why Modal and not OpenAI?" → the matrix answers with numbers.
• You need to re-evaluate because a price changed → you recalculate the score, you don't redo the debate.
• A new team member joins → they understand the decision by reading the matrix, not by asking.

It's the difference between a technical decision and a group opinion with a technical appearance.

Step 1 — Distinguish constraints from preferences

Before weighting anything, separate what's non-negotiable vs what's desirable:

Filtering first avoids wasting time scoring options that aren't viable. If your client requires hosting in the EU, there's no point scoring OpenAI (US) — you eliminate it beforehand.

Step 2 — Define weights

Sum of weights = 1.0. Typical distribution for different products:

There are no "universally correct weights". There are weights that reflect what matters to your product. Discuss them with your team before scoring.

Step 3 — Normalize the metrics

Latency (seconds), cost ($), quality (% score) are different units. They can't be added directly. Normalize each one to [0, 1] where 1 = best:

latency_norm  = 1 - (latency_provider - latency_min) / (latency_max - latency_min)
cost_norm     = 1 - (cost_provider - cost_min) / (cost_max - cost_min)
quality_norm  = (quality_provider - quality_min) / (quality_max - quality_min)

The 1 - trick in latency and cost is because "lower is better" in those dimensions, but we want "best = 1" in all of them.

Step 4 — Calculate the composite score

score = (latency_norm × weight_latency)
      + (cost_norm × weight_cost)
      + (quality_norm × weight_quality)

Higher score = better option according to your weights.

Implementation: reusable matrix

Create decision_matrix.py:

# decision_matrix.py
from dataclasses import dataclass
from typing import Literal

@dataclass
class Provider:
    name: str
    latency_p95_s: float          # from your benchmark
    cost_monthly_usd: float       # for your projected volume
    quality_score: float          # 0-100 (% score from your rubric)
    meets_data_residency_eu: bool = True
    meets_soc2: bool = True


@dataclass
class Weights:
    latency: float = 0.33
    cost: float = 0.33
    quality: float = 0.34

    def __post_init__(self):
        total = self.latency + self.cost + self.quality
        assert abs(total - 1.0) < 0.001, f"Weights must sum to 1.0, they sum to {total}"


@dataclass
class Constraints:
    data_residency_eu: bool = False
    soc2: bool = False
    budget_max_usd: float = float("inf")


def filter_by_constraints(
    providers: list[Provider], constraints: Constraints
) -> tuple[list[Provider], list[str]]:
    """Filters providers that don't meet absolute constraints. Returns (valid, discarded)."""
    valid = []
    discarded = []
    for p in providers:
        reasons = []
        if constraints.data_residency_eu and not p.meets_data_residency_eu:
            reasons.append("fails EU data residency")
        if constraints.soc2 and not p.meets_soc2:
            reasons.append("fails SOC2")
        if p.cost_monthly_usd > constraints.budget_max_usd:
            reasons.append(f"exceeds budget (${p.cost_monthly_usd:.0f} > ${constraints.budget_max_usd:.0f})")
        if reasons:
            discarded.append(f"{p.name}: {', '.join(reasons)}")
        else:
            valid.append(p)
    return valid, discarded


def normalize(values: list[float], lower_is_better: bool = False) -> list[float]:
    """Normalizes to [0, 1] where 1 = best."""
    if not values or max(values) == min(values):
        return [1.0] * len(values)
    vmin, vmax = min(values), max(values)
    if lower_is_better:
        return [1 - (v - vmin) / (vmax - vmin) for v in values]
    else:
        return [(v - vmin) / (vmax - vmin) for v in values]


def calculate_scores(providers: list[Provider], weights: Weights) -> list[tuple[Provider, float, dict]]:
    latencies = [p.latency_p95_s for p in providers]
    costs = [p.cost_monthly_usd for p in providers]
    qualities = [p.quality_score for p in providers]

    n_lat = normalize(latencies, lower_is_better=True)
    n_cost = normalize(costs, lower_is_better=True)
    n_qual = normalize(qualities, lower_is_better=False)

    results = []
    for p, nl, nc, nq in zip(providers, n_lat, n_cost, n_qual):
        score = nl * weights.latency + nc * weights.cost + nq * weights.quality
        breakdown = {
            "latency_norm": nl,
            "cost_norm": nc,
            "quality_norm": nq,
            "contribution_latency": nl * weights.latency,
            "contribution_cost": nc * weights.cost,
            "contribution_quality": nq * weights.quality,
        }
        results.append((p, score, breakdown))
    return sorted(results, key=lambda x: x[1], reverse=True)


def report(providers: list[Provider], weights: Weights, constraints: Constraints):
    print(f"\n=== Applied constraints ===")
    print(f"  data residency EU: {constraints.data_residency_eu}")
    print(f"  SOC2: {constraints.soc2}")
    print(f"  Budget max: ${constraints.budget_max_usd:,.0f}/month")

    valid, discarded = filter_by_constraints(providers, constraints)

    if discarded:
        print(f"\n=== Discarded by constraints ===")
        for d in discarded:
            print(f"  ✗ {d}")

    if not valid:
        print("\n⚠️  NO provider meets the constraints. Review.")
        return

    print(f"\n=== Weights ===")
    print(f"  Latency: {weights.latency:.2f}")
    print(f"  Cost:    {weights.cost:.2f}")
    print(f"  Quality: {weights.quality:.2f}")

    scores = calculate_scores(valid, weights)

    print(f"\n=== Ranking ===\n")
    print(f"{'Provider':<35} {'P95':>7} {'$/mo':>10} {'Qual%':>6} {'Score':>8}")
    print("-" * 70)
    for p, score, _ in scores:
        print(f"{p.name:<35} {p.latency_p95_s:>6.2f}s {p.cost_monthly_usd:>9.0f} {p.quality_score:>5.1f} {score:>8.3f}")

    winner, score_w, breakdown = scores[0]
    print(f"\n→ Recommendation: {winner.name} (score {score_w:.3f})")
    print(f"  Score breakdown:")
    print(f"    Latency: {breakdown['contribution_latency']:.3f}")
    print(f"    Cost:    {breakdown['contribution_cost']:.3f}")
    print(f"    Quality: {breakdown['contribution_quality']:.3f}")


# ============================================================
# Worked example
# ============================================================
if __name__ == "__main__":
    # Hypothetical data for your case (replace with your real measurements)
    providers = [
        Provider("OpenAI GPT-4o-mini", latency_p95_s=2.3, cost_monthly_usd=39, quality_score=86,
                 meets_data_residency_eu=False),
        Provider("OpenAI GPT-4o", latency_p95_s=3.1, cost_monthly_usd=900, quality_score=92,
                 meets_data_residency_eu=False),
        Provider("OpenRouter Mistral 7B", latency_p95_s=3.0, cost_monthly_usd=10, quality_score=69,
                 meets_data_residency_eu=False),   # aggregator, assumed non-EU
        Provider("Modal Mistral 7B + warm pool", latency_p95_s=2.7, cost_monthly_usd=283, quality_score=69,
                 meets_data_residency_eu=False),   # serverless US, assumed non-EU
        Provider("Self-hosted Ollama Mistral EU", latency_p95_s=5.9, cost_monthly_usd=720, quality_score=69,
                 meets_data_residency_eu=True),
    ]

    # Case A: real-time consumer chatbot (latency matters)
    weights_consumer = Weights(latency=0.5, cost=0.2, quality=0.3)
    constraints_free = Constraints()

    print("\n" + "=" * 70)
    print("CASE A — Real-time consumer chatbot, no special constraints")
    print("=" * 70)
    report(providers, weights_consumer, constraints_free)

    # Case B: EU enterprise client
    weights_enterprise = Weights(latency=0.3, cost=0.2, quality=0.5)
    constraints_eu = Constraints(data_residency_eu=True, budget_max_usd=1000)

    print("\n" + "=" * 70)
    print("CASE B — Enterprise client, EU data residency required, max $1k/month")
    print("=" * 70)
    report(providers, weights_enterprise, constraints_eu)

Typical reading of results

======================================================================
CASE A — Real-time consumer chatbot, no special constraints
======================================================================

=== Weights ===
  Latency: 0.50
  Cost:    0.20
  Quality: 0.30

=== Ranking ===

Provider                             P95     $/mo   Qual%  Score
----------------------------------------------------------------------
OpenAI GPT-4o-mini                  2.30s        39  86.0   0.915
OpenAI GPT-4o                       3.10s       900  92.0   0.689
OpenRouter Mistral 7B               3.00s        10  69.0   0.603
Modal Mistral 7B + warm pool        2.70s       283  69.0   0.583
Self-hosted Ollama Mistral EU       5.90s       720  69.0   0.040

→ Recommendation: OpenAI GPT-4o-mini (score 0.915)
  Breakdown:
    Latency: 0.500  (perfect: it's the fastest)
    Cost:    0.193  (nearly the best among costs)
    Quality: 0.222  (good, not the best)

======================================================================
CASE B — Enterprise client, EU data residency required
======================================================================

=== Discarded by constraints ===
  ✗ OpenAI GPT-4o-mini: fails EU data residency
  ✗ OpenAI GPT-4o: fails EU data residency
  ✗ OpenRouter Mistral 7B: fails EU data residency
  ✗ Modal Mistral 7B + warm pool: fails EU data residency

⚠️  Only remains: Self-hosted Ollama Mistral EU

Reading:

• Case A: OpenAI GPT-4o-mini wins clearly. Competitive latency and cost, very good quality.
• Case B: The constraints drastically reduce the space. Only Self-hosted EU remains. The choice becomes obvious, not by benchmark but by constraint.

Sensitivity analysis

Critical question: how sensitive is the decision to your weights?

Change the weights slightly and see whether the ranking changes. If small variations change the winner, the decision is fragile — you need more data before committing.

# Sensitivity analysis
for weight_lat in [0.3, 0.4, 0.5, 0.6]:
    weight_cost = (1 - weight_lat) / 2
    weight_qual = (1 - weight_lat) / 2
    weights = Weights(latency=weight_lat, cost=weight_cost, quality=weight_qual)
    scores = calculate_scores(valid, weights)
    print(f"\n  Latency={weight_lat}: winner = {scores[0][0].name}")

If you see "winner = OpenAI" in every case, the decision is robust. If you see it alternate, it's worth digging deeper.

Pattern: a table of reasons, not just numbers

A numeric matrix hides the why. To document an enterprise decision, complement it with a table of reasons:

That table is what you paste into the decision doc.

Common traps

Trap 1 — "The CTO's weights, not the product's." Sometimes the CTO favors cost out of a limited-funds bias, but the product needs quality. Discuss the weights with product/users, not just engineering.

Trap 2 — "Absolute score, not relative." A score of 0.83 vs 0.81 is almost a tie. Treating them as a clear difference is noise. If the top 2 are within <0.05 of each other, decide with qualitative criteria (team familiarity, support, ecosystem).

Trap 3 — "Normalization dramatizes small differences." If latencies are 2.0s, 2.1s, 2.2s, normalization turns them into 1.0, 0.5, 0.0. That difference looks huge in the score but is imperceptible to the user (100ms). Consider using normalization with a clip or threshold for minimal differences.

Trap 4 — "You forgot a key dimension." Latency + cost + quality is a good default, but there are others: vendor lock-in, ecosystem, support quality, alignment with evolving compliance. Add them if your case warrants it.

Trap 5 — "A decision set in stone." Pricing changes. New models come out every quarter. Your matrix should be reviewed quarterly, not decided once and forgotten.

Exercise

Build the matrix for your own case:

1. Define your absolute constraints (data residency, budget, compliance)
2. Weight latency/cost/quality according to your product's real priorities
3. Plug in the numbers you measured in capsules 02-04
4. Calculate scores
5. Do a sensitivity analysis changing weights ±0.1
6. Document your recommendation with the table of reasons

Product: B2B technical support assistant
Absolute constraints:
  - data_residency_eu for enterprise clients (5 clients)
  - budget_max: $2000/month
Weights:
  - Latency: 0.3 (important but not critical)
  - Cost: 0.2 (relevant)
  - Quality: 0.5 (key for a B2B product)

After filters: 2 valid providers
  - Modal with warm pool in EU region (if certified)
  - Self-hosted EU

Score:
  - Modal EU: 0.74 (latency OK, quality OK, medium cost)
  - Self-hosted: 0.51 (worse latency, high cost)

Recommendation: Modal EU with a migration plan to self-hosted if volume exceeds 500K req/month.

Sensitivity: the decision holds with weights in the range [latency 0.2-0.4, cost 0.1-0.3, quality 0.4-0.6]. Robust decision.

Summary

You learned:

• ✅ Separate absolute constraints (filter) from preferences (weight)
• ✅ Assign weights according to the product's real priorities, not opinion
• ✅ Normalize metrics on different scales to make them comparable
• ✅ Calculate a composite score that ranks defensibly
• ✅ Do a sensitivity analysis to validate robustness
• ✅ Document with a table of reasons, not just numbers

Checkpoint: if you can produce a ranking with a score, a justification of the weights, and "what changes if I modify the weights", you're ready.

Next capsule

06 — Migration paths. Your matrix said "X wins today". Tomorrow something changes (a price, a new model, a legal restriction) and you need to migrate. How much code changes to go from OpenAI to OpenRouter? What about Modal to Ollama? We look at the real paths and their costs.

Resources

1. Multi-Criteria Decision Analysis (MCDA) — overview — general theory.
2. TOPSIS method — alternative scoring method (similarity to the ideal).
3. Architectural decision records (ADR) — pattern for documenting technical decisions.
4. Wardley Mapping — to place providers in the maturity cycle.