Module 6: Decision Matrix for AI Engineers

Capsule 03: Weights, Scoring, and Matrix Methodology

Master the quantitative mechanics of the decision matrix: how to assign weights that reflect real priorities, how to score providers consistently, and how to normalize results so the comparison is fair.

By the end of this capsule:

✅ You'll assign weights using techniques that eliminate bias
✅ You'll build objective scoring rubrics per criterion
✅ You'll normalize scores to compare different dimensions
✅ You'll implement the complete scoring formula in Python

Estimated time: 25-35 minutes

Capsule description

In the previous capsule you defined WHAT to evaluate. Now you need to define HOW to evaluate it. The difference between a solid decision and a decision disguised as analysis is in the scoring methodology. If you assign arbitrary weights or score "by eye", your sophisticated matrix produces exactly the same result as choosing at random — but with the false confidence of a number.

This capsule teaches you three professional techniques for assigning weights (fixed distribution, pairwise comparison, and stack ranking), two scoring scales with concrete rubrics for each level, and the normalization process that lets you compare apples with oranges (latency in milliseconds vs cost in dollars vs operational complexity).

In the end, you'll have a complete ScoringEngine in Python that you can feed with your criteria and providers to get a reproducible quantitative ranking. It's not magic — it's discipline.

Why weights matter more than scores

Consider this example:

# Same provider, different weights = different winner

provider_scores = {
    "pinecone": {"latency": 0.9, "cost": 0.5, "ops_simplicity": 1.0},
    "qdrant_self": {"latency": 0.8, "cost": 0.9, "ops_simplicity": 0.4},
}

# Team A: startup without DevOps, speed matters
weights_startup = {"latency": 3, "cost": 4, "ops_simplicity": 5}

# Team B: company with an SRE, cost dominates
weights_enterprise = {"latency": 5, "cost": 5, "ops_simplicity": 2}

def weighted_score(scores: dict, weights: dict) -> float:
    total = sum(weights[k] * scores[k] for k in weights)
    max_possible = sum(weights.values())
    return round((total / max_possible) * 100, 1)

print("=== Team A (startup without DevOps) ===")
for name, scores in provider_scores.items():
    print(f"  {name}: {weighted_score(scores, weights_startup)}%")

print("\n=== Team B (enterprise with SRE) ===")
for name, scores in provider_scores.items():
    print(f"  {name}: {weighted_score(scores, weights_enterprise)}%")

=== Team A (startup without DevOps) ===
  pinecone: 80.8%
  qdrant_self: 66.7%

=== Team B (enterprise with SRE) ===
  pinecone: 75.0%
  qdrant_self: 77.5%

Same data, different conclusion. The weights determine the result more than the individual scores. That's why you need a rigorous process to assign them.

Technique 1: Fixed point distribution

Each stakeholder gets a fixed budget of points (for example, 100) and distributes them among the criteria. This forces real trade-offs:

def fixed_distribution(criteria: list[str], total_points: int = 100) -> dict:
    """
    Simulate the fixed point distribution.
    In practice, each stakeholder fills this in individually.
    """
    print(f"Distribute {total_points} points among {len(criteria)} criteria.")
    print("Rule: the sum MUST be exactly {total_points}.")
    print("Criteria:", ", ".join(criteria))

    return {}  # In practice, filled in manually


def aggregate_distributions(distributions: list[dict]) -> dict:
    """Average the distributions of multiple stakeholders."""
    all_keys = set()
    for d in distributions:
        all_keys.update(d.keys())

    aggregated = {}
    for key in all_keys:
        values = [d.get(key, 0) for d in distributions]
        aggregated[key] = round(sum(values) / len(values), 1)

    return aggregated


# Example: 3 stakeholders distribute 100 points
cto_weights = {
    "latency": 25, "scale": 25, "cost": 15,
    "ops_simplicity": 10, "sdk_quality": 15, "compliance": 10
}
pm_weights = {
    "latency": 15, "scale": 10, "cost": 30,
    "ops_simplicity": 20, "sdk_quality": 5, "compliance": 20
}
dev_weights = {
    "latency": 20, "scale": 15, "cost": 10,
    "ops_simplicity": 15, "sdk_quality": 30, "compliance": 10
}

# Validation: each distribution must add up to 100
for name, weights in [("CTO", cto_weights), ("PM", pm_weights), ("Dev", dev_weights)]:
    total = sum(weights.values())
    status = "✅" if total == 100 else f"❌ (sum {total})"
    print(f"{name}: {status}")

# Aggregation
final_weights = aggregate_distributions([cto_weights, pm_weights, dev_weights])
print("\nFinal weights (average):")
for criterion, weight in sorted(final_weights.items(), key=lambda x: -x[1]):
    print(f"  {criterion}: {weight}")

CTO: ✅
PM: ✅
Dev: ✅

Final weights (average):
  latency: 20.0
  cost: 18.3
  sdk_quality: 16.7
  scale: 16.7
  ops_simplicity: 15.0
  compliance: 13.3

Advantage: Forces real trade-offs (you can't say "everything is important"). Disadvantage: Doesn't capture the relative intensity between pairs of criteria.

Technique 2: Pairwise comparison

Compare each criterion against every other criterion and decide which is more important. The count of "wins" determines the weight:

from itertools import combinations


def pairwise_comparison(criteria: list[str], preferences: dict[tuple, str]) -> dict:
    """
    Calculate weights based on pairwise comparison.

    Args:
        criteria: List of criterion names
        preferences: Dict of (criterion_a, criterion_b) -> winner
    """
    wins = {c: 0 for c in criteria}

    for (a, b), winner in preferences.items():
        wins[winner] += 1

    total_wins = sum(wins.values())
    if total_wins == 0:
        return {c: 1.0 / len(criteria) for c in criteria}

    weights = {c: round(w / total_wins, 3) for c, w in wins.items()}
    return weights


criteria = ["latency", "cost", "ops", "sdk", "scale", "compliance"]

# For each pair, which is more important for YOUR project?
preferences = {
    ("latency", "cost"): "latency",
    ("latency", "ops"): "latency",
    ("latency", "sdk"): "latency",
    ("latency", "scale"): "scale",
    ("latency", "compliance"): "latency",
    ("cost", "ops"): "cost",
    ("cost", "sdk"): "cost",
    ("cost", "scale"): "scale",
    ("cost", "compliance"): "compliance",
    ("ops", "sdk"): "ops",
    ("ops", "scale"): "scale",
    ("ops", "compliance"): "compliance",
    ("sdk", "scale"): "scale",
    ("sdk", "compliance"): "sdk",
    ("scale", "compliance"): "scale",
}

weights = pairwise_comparison(criteria, preferences)
print("Weights by pairwise comparison:")
for criterion, weight in sorted(weights.items(), key=lambda x: -x[1]):
    print(f"  {criterion}: {weight:.1%}")

Weights by pairwise comparison:
  scale: 33.3%
  latency: 26.7%
  cost: 13.3%
  compliance: 13.3%
  ops: 6.7%
  sdk: 6.7%

Advantage: More intuitive ("what matters more, A or B?") and produces clear rankings. Disadvantage: With N criteria, you need N×(N-1)/2 comparisons (15 for 6 criteria).

Technique 3: Stack ranking

The simplest: order all criteria from most to least important and assign decreasing weights automatically:

def stack_rank_weights(ranked_criteria: list[str], method: str = "linear") -> dict:
    """
    Assign weights based on ordinal ranking.

    Args:
        ranked_criteria: Ordered list (most important first)
        method: 'linear' (N, N-1, ..., 1) or 'geometric' (2^N, 2^(N-1), ...)
    """
    n = len(ranked_criteria)

    if method == "linear":
        raw_weights = {c: n - i for i, c in enumerate(ranked_criteria)}
    elif method == "geometric":
        raw_weights = {c: 2 ** (n - 1 - i) for i, c in enumerate(ranked_criteria)}
    else:
        raise ValueError(f"Unknown method: {method}")

    total = sum(raw_weights.values())
    normalized = {c: round(w / total, 3) for c, w in raw_weights.items()}
    return normalized


my_ranking = ["scale", "latency", "cost", "ops", "compliance", "sdk"]

print("Stack rank - Linear:")
linear = stack_rank_weights(my_ranking, "linear")
for c, w in linear.items():
    print(f"  {c}: {w:.1%}")

print("\nStack rank - Geometric:")
geometric = stack_rank_weights(my_ranking, "geometric")
for c, w in geometric.items():
    print(f"  {c}: {w:.1%}")

Stack rank - Linear:
  scale: 28.6%
  latency: 23.8%
  cost: 19.0%
  ops: 14.3%
  compliance: 9.5%
  sdk: 4.8%

Stack rank - Geometric:
  scale: 50.8%
  latency: 25.4%
  cost: 12.7%
  ops: 6.3%
  compliance: 3.2%
  sdk: 1.6%

Linear distributes weights gradually. Geometric drastically amplifies the difference between the first and last criterion. Use geometric only when your #1 criterion completely dominates the decision.

Scoring scales: concrete rubrics

A score without a rubric is an opinion in a numerical disguise. Define exactly what each level means:

5-level scale (recommended)

scoring_rubric = {
    1.0: {
        "label": "Fully meets",
        "definition": "Exceeds the ideal threshold. No relevant trade-offs.",
        "evidence_required": "Benchmark/docs proving the ideal threshold is met"
    },
    0.75: {
        "label": "Meets well",
        "definition": "Meets the minimum threshold and approaches the ideal. Minor trade-off.",
        "evidence_required": "Benchmark/docs + a note on the trade-off"
    },
    0.5: {
        "label": "Partial compliance",
        "definition": "Meets the minimum threshold but far from the ideal. Significant trade-off.",
        "evidence_required": "Document the gap and a mitigation plan"
    },
    0.25: {
        "label": "Weak compliance",
        "definition": "Doesn't meet the minimum threshold but is close. High risk.",
        "evidence_required": "Document the risk and the necessary workaround"
    },
    0.0: {
        "label": "Doesn't meet",
        "definition": "Doesn't meet the minimum threshold. No viable mitigation plan.",
        "evidence_required": "Mark as a deal-breaker if the criterion is a MUST"
    }
}

for score, details in scoring_rubric.items():
    print(f"\n{score} — {details['label']}")
    print(f"  Definition: {details['definition']}")
    print(f"  Evidence: {details['evidence_required']}")

Dimension-specific rubrics

The meaning of "0.75" depends on the criterion. Here are concrete rubrics:

dimension_rubrics = {
    "latency_p95": {
        1.0: "< 100ms (exceeds the real-time expectation)",
        0.75: "100-250ms (good for chatbots/search)",
        0.5: "250-500ms (acceptable for batch/internal)",
        0.25: "500ms-1s (problematic for UX)",
        0.0: "> 1s (unacceptable for production)"
    },
    "cost_monthly": {
        1.0: "< 50% of the budget (ample margin)",
        0.75: "50-80% of the budget (viable)",
        0.5: "80-100% of the budget (no margin)",
        0.25: "100-150% of the budget (requires negotiation)",
        0.0: "> 150% of the budget (out of range)"
    },
    "ops_complexity": {
        1.0: "Managed, < 2h/month maintenance",
        0.75: "Semi-managed, 2-5h/month",
        0.5: "Simple self-hosted, 5-10h/month",
        0.25: "Self-hosted cluster, 10-20h/month",
        0.0: "Dedicated operations, > 20h/month or SRE required"
    },
    "sdk_quality": {
        1.0: "Mature SDK, type hints, async, testing mode, excellent docs",
        0.75: "Working SDK, good docs, minor issues",
        0.5: "Basic SDK, incomplete docs, workarounds needed",
        0.25: "Immature SDK, frequent bugs, minimal docs",
        0.0: "No SDK in your language or an abandoned SDK"
    },
    "community": {
        1.0: "> 10K stars, active, SO coverage, complete integrations",
        0.75: "5-10K stars, active, main integrations",
        0.5: "1-5K stars, moderate activity, partial integrations",
        0.25: "< 1K stars, low activity, few integrations",
        0.0: "New/abandoned project, no visible community"
    },
    "compliance": {
        1.0: "SOC2 + GDPR + HIPAA + CMEK + audit logs",
        0.75: "SOC2 + GDPR + encryption at rest",
        0.5: "Encryption at rest + basic auth",
        0.25: "Only basic auth, no certifications",
        0.0: "No encryption or certifications"
    }
}

print("=== Rubric: p95 latency ===")
for score, description in dimension_rubrics["latency_p95"].items():
    print(f"  {score}: {description}")

Normalization: comparing different dimensions

Latency is measured in milliseconds, cost in dollars, ops in hours. For the weights to work, you need to normalize everything to a common scale (0-1):

def normalize_score(
    raw_value: float,
    min_acceptable: float,
    ideal_value: float,
    lower_is_better: bool = True
) -> float:
    """
    Normalize a raw value to a 0-1 scale.

    Args:
        raw_value: The provider's measured value
        min_acceptable: Minimum acceptable threshold
        ideal_value: Ideal value (1.0)
        lower_is_better: True for latency/cost, False for throughput/recall
    """
    if lower_is_better:
        if raw_value <= ideal_value:
            return 1.0
        elif raw_value >= min_acceptable:
            return 0.25
        else:
            range_size = min_acceptable - ideal_value
            if range_size == 0:
                return 1.0 if raw_value <= ideal_value else 0.0
            normalized = 1.0 - ((raw_value - ideal_value) / range_size)
            return max(0.0, min(1.0, round(normalized, 2)))
    else:
        if raw_value >= ideal_value:
            return 1.0
        elif raw_value <= min_acceptable:
            return 0.25
        else:
            range_size = ideal_value - min_acceptable
            if range_size == 0:
                return 1.0 if raw_value >= ideal_value else 0.0
            normalized = (raw_value - min_acceptable) / range_size
            return max(0.0, min(1.0, round(normalized, 2)))


# Normalization examples
print("=== Latency normalization (lower is better) ===")
latencies = [50, 100, 200, 350, 500, 800]
for lat in latencies:
    score = normalize_score(lat, min_acceptable=500, ideal_value=100, lower_is_better=True)
    print(f"  {lat}ms → {score:.2f}")

print("\n=== Recall normalization (higher is better) ===")
recalls = [0.80, 0.90, 0.95, 0.98, 1.0]
for recall in recalls:
    score = normalize_score(recall, min_acceptable=0.90, ideal_value=0.98, lower_is_better=False)
    print(f"  {recall:.0%} → {score:.2f}")

=== Latency normalization (lower is better) ===
  50ms → 1.00
  100ms → 1.00
  200ms → 0.75
  350ms → 0.38
  500ms → 0.25
  800ms → 0.25

=== Recall normalization (higher is better) ===
  80% → 0.25
  90% → 0.25
  95% → 0.62
  98% → 1.00
  100% → 1.00

ScoringEngine: complete implementation

Here's the class that integrates weights, scoring, and normalization:

from dataclasses import dataclass, field


@dataclass
class CriterionConfig:
    name: str
    weight: float
    min_acceptable: float
    ideal_value: float
    lower_is_better: bool = True
    unit: str = ""


@dataclass
class ProviderScore:
    name: str
    raw_scores: dict[str, float] = field(default_factory=dict)
    normalized_scores: dict[str, float] = field(default_factory=dict)
    weighted_scores: dict[str, float] = field(default_factory=dict)
    total_score: float = 0.0
    total_percentage: float = 0.0


class ScoringEngine:
    def __init__(self, criteria: list[CriterionConfig]):
        self.criteria = {c.name: c for c in criteria}
        self.providers: dict[str, ProviderScore] = {}

    def add_provider(self, name: str, raw_scores: dict[str, float]):
        provider = ProviderScore(name=name, raw_scores=raw_scores)

        for criterion_name, raw_value in raw_scores.items():
            config = self.criteria[criterion_name]

            normalized = normalize_score(
                raw_value,
                config.min_acceptable,
                config.ideal_value,
                config.lower_is_better
            )
            provider.normalized_scores[criterion_name] = normalized

            weighted = normalized * config.weight
            provider.weighted_scores[criterion_name] = round(weighted, 2)

        provider.total_score = sum(provider.weighted_scores.values())
        max_possible = sum(c.weight for c in self.criteria.values())
        provider.total_percentage = round(
            (provider.total_score / max_possible) * 100, 1
        )

        self.providers[name] = provider

    def ranking(self) -> list[ProviderScore]:
        return sorted(
            self.providers.values(),
            key=lambda p: p.total_score,
            reverse=True
        )

    def sensitivity_analysis(self, criterion_name: str, weight_range: list[float]) -> dict:
        """How does the ranking change if you change a criterion's weight?"""
        results = {}
        original_weight = self.criteria[criterion_name].weight

        for new_weight in weight_range:
            self.criteria[criterion_name].weight = new_weight
            for name, provider in self.providers.items():
                raw = provider.raw_scores
                self.add_provider(name, raw)

            ranking = [(p.name, p.total_percentage) for p in self.ranking()]
            results[new_weight] = ranking

        self.criteria[criterion_name].weight = original_weight
        for name, provider in self.providers.items():
            self.add_provider(name, provider.raw_scores)

        return results

    def report(self) -> str:
        lines = ["=== SCORING REPORT ===\n"]

        header = f"{'Criterion':<20} {'Weight':<6}"
        for p in self.ranking():
            header += f" {p.name:<15}"
        lines.append(header)
        lines.append("-" * len(header))

        for criterion_name, config in self.criteria.items():
            row = f"{criterion_name:<20} {config.weight:<6.1f}"
            for p in self.ranking():
                raw = p.raw_scores.get(criterion_name, 0)
                norm = p.normalized_scores.get(criterion_name, 0)
                row += f" {raw:>5.1f}→{norm:.2f}    "
            lines.append(row)

        lines.append("-" * len(header))
        total_row = f"{'TOTAL':<20} {'':6}"
        for p in self.ranking():
            total_row += f" {p.total_percentage:>8.1f}%     "
        lines.append(total_row)

        return "\n".join(lines)


# Complete example
engine = ScoringEngine([
    CriterionConfig("latency_p95", weight=5.0, min_acceptable=500,
                    ideal_value=100, lower_is_better=True, unit="ms"),
    CriterionConfig("cost_monthly", weight=4.0, min_acceptable=500,
                    ideal_value=100, lower_is_better=True, unit="USD"),
    CriterionConfig("ops_hours", weight=4.0, min_acceptable=15,
                    ideal_value=2, lower_is_better=True, unit="h/month"),
    CriterionConfig("recall", weight=3.0, min_acceptable=0.90,
                    ideal_value=0.98, lower_is_better=False, unit="%"),
    CriterionConfig("sdk_score", weight=2.0, min_acceptable=0.5,
                    ideal_value=0.9, lower_is_better=False, unit="0-1"),
])

engine.add_provider("Pinecone", {
    "latency_p95": 120, "cost_monthly": 350,
    "ops_hours": 2, "recall": 0.96, "sdk_score": 0.85
})

engine.add_provider("Qdrant Cloud", {
    "latency_p95": 150, "cost_monthly": 200,
    "ops_hours": 3, "recall": 0.95, "sdk_score": 0.80
})

engine.add_provider("ChromaDB", {
    "latency_p95": 300, "cost_monthly": 60,
    "ops_hours": 10, "recall": 0.92, "sdk_score": 0.75
})

engine.add_provider("Weaviate Cloud", {
    "latency_p95": 180, "cost_monthly": 280,
    "ops_hours": 2, "recall": 0.94, "sdk_score": 0.70
})

print(engine.report())
print("\n=== RANKING ===")
for i, p in enumerate(engine.ranking(), 1):
    print(f"  #{i} {p.name}: {p.total_percentage}%")

Interpreting results

Don't take the result as an absolute verdict. Use these guidelines:

def interpret_score(percentage: float) -> dict:
    """Interpret a provider's final score."""
    if percentage >= 85:
        return {
            "verdict": "Excellent fit",
            "action": "Proceed with an immediate PoC",
            "confidence": "High",
            "risk": "Low"
        }
    elif percentage >= 70:
        return {
            "verdict": "Good fit with trade-offs",
            "action": "Proceed with a PoC, document mitigations",
            "confidence": "Medium-high",
            "risk": "Moderate — monitor the criteria where it scored low"
        }
    elif percentage >= 55:
        return {
            "verdict": "Partial fit",
            "action": "Only if there's no better alternative or a dominant constraint",
            "confidence": "Medium",
            "risk": "High — requires an explicit mitigation plan"
        }
    else:
        return {
            "verdict": "Not recommended",
            "action": "Discard except in exceptional circumstances",
            "confidence": "Low",
            "risk": "Very high"
        }

# Gap analysis: where does each provider lose points?
def gap_analysis(provider: ProviderScore, criteria: dict[str, CriterionConfig]) -> list[dict]:
    """Identify the criteria where a provider loses the most points."""
    gaps = []
    for name, config in criteria.items():
        normalized = provider.normalized_scores.get(name, 0)
        if normalized < 0.75:
            lost_points = (1.0 - normalized) * config.weight
            gaps.append({
                "criterion": name,
                "normalized_score": normalized,
                "weight": config.weight,
                "points_lost": round(lost_points, 2),
                "raw_value": provider.raw_scores.get(name, 0)
            })
    return sorted(gaps, key=lambda g: -g["points_lost"])

Sensitivity analysis

Does your result change if you adjust a weight? If so, your decision is fragile:

def sensitivity_check(engine: ScoringEngine, criterion: str) -> None:
    """Check whether the ranking is stable against weight changes."""
    print(f"\n=== Sensitivity: what happens if I change the weight of '{criterion}'? ===")

    results = engine.sensitivity_analysis(criterion, [1.0, 2.0, 3.0, 4.0, 5.0])
    for weight, ranking in results.items():
        leader = ranking[0]
        second = ranking[1]
        gap = leader[1] - second[1]
        stability = "🟢 stable" if gap > 5 else "🟡 close" if gap > 2 else "🔴 fragile"
        print(f"  Weight {weight}: #{1} {leader[0]} ({leader[1]}%) "
              f"vs #{2} {second[0]} ({second[1]}%) — gap: {gap:.1f}% {stability}")

# Example
sensitivity_check(engine, "cost_monthly")

If changing a weight from 3 to 5 changes the winner, your decision depends on that criterion. That's not wrong — but you must be aware of it.

Bias in scoring: how to detect and reduce it

bias_checklist = {
    "anchoring_bias": {
        "description": "The first provider you evaluated biases the others",
        "detection": "Did you score the first one higher on almost everything?",
        "mitigation": "Score all providers on one criterion before moving to the next"
    },
    "familiarity_bias": {
        "description": "You score the provider you already know higher",
        "detection": "Does the one you currently use win by a lot?",
        "mitigation": "Include a team member who has NOT used the current provider"
    },
    "recency_bias": {
        "description": "The latest blog post or tweet influences your score",
        "detection": "Did your evaluation change after reading an article?",
        "mitigation": "Use only evidence from official docs, benchmarks, and your own PoCs"
    },
    "halo_effect": {
        "description": "An excellent criterion inflates the others",
        "detection": "Does a provider have almost everything at 1.0?",
        "mitigation": "Each criterion is scored with its own rubric, not by 'general feeling'"
    },
    "sunk_cost_bias": {
        "description": "You already invested time in a provider and don't want to discard it",
        "detection": "Are you looking for justifications to keep the current one?",
        "mitigation": "Evaluate as if starting from scratch (greenfield analysis)"
    }
}

print("=== Bias checklist ===")
for bias, info in bias_checklist.items():
    print(f"\n⚠️ {bias}")
    print(f"  Description: {info['description']}")
    print(f"  Detection: {info['detection']}")
    print(f"  Mitigation: {info['mitigation']}")

🔧 Troubleshooting

Problem 1: "Everything comes out tied"

Symptom: Two or more providers have scores within 3-5% of each other.

Solution: Increase the resolution in the criteria that weigh the most. If "latency" has weight 5, subdivide it into latency_p50, latency_p95, latency_p99. Another option: add a "tiebreaker" criterion like migration ease or the team's previous experience.

Problem 2: "The result contradicts the team's intuition"

Symptom: The matrix's winner isn't the one the team "feels" should win.

Solution: Don't dismiss the intuition — check whether the weights reflect the real priorities. Sometimes intuition captures information that isn't in the matrix (a previous negative experience, a relationship with the vendor). If, after reviewing the weights, the numbers still contradict it, document the override and its justification.

Problem 3: "I don't have data to score objectively"

Symptom: You haven't done benchmarks or PoCs and you're scoring based on marketing.

Solution: Use a provisional score (mark it with ⚠️) based on documentation and public benchmarks. Plan a 2-3 day PoC for the top 2 providers to validate the provisional scores before deciding.

Problem 4: "Each stakeholder produces a completely different ranking"

Symptom: The CTO chooses Pinecone, the PM chooses ChromaDB, the Dev chooses Qdrant.

Solution: This reveals different priorities, not a methodology problem. Aggregate the weights (average) and only discuss the criteria where the divergence is greater than 2 points. The disagreement IS valuable information.

Problem 5: "A provider wins because of the weights, not because it's better"

Symptom: If you change the weights slightly, another provider wins.

Solution: Do a sensitivity analysis. If the result changes with ±1 variations in a weight, the decision is fragile. In that case, prioritize the provider with the lowest operational risk as a tiebreaker.

🏋️ Exercises

Exercise 1: Fixed point distribution

Distribute 100 points among these 6 criteria for an internal RAG project (mid-size company, 3 developers, no SRE, 500K expected vectors):

Latency
Cost
Operational complexity
SDK quality
Community
Compliance

Solution

# For internal RAG, mid-size company, no SRE, 500K vectors:
my_distribution = {
    "latency": 15,       # Internal: moderate tolerance
    "cost": 20,          # Limited budget
    "ops_complexity": 30, # No SRE: OPS dominates the decision
    "sdk_quality": 15,   # 3 devs = SDK matters
    "community": 10,     # Important but not critical
    "compliance": 10,    # Mid-size company: basic requirements
}

assert sum(my_distribution.values()) == 100, "Must add up to 100"

print("Distribution:")
for criterion, points in sorted(my_distribution.items(), key=lambda x: -x[1]):
    bar = "█" * (points // 2)
    print(f"  {criterion:<20} {points:>3} pts {bar}")

# Ops dominates because without an SRE, every hour of maintenance
# is an hour a developer isn't building features

Exercise 2: Pairwise comparison

Do the 15 pairwise comparisons for the 6 criteria from the previous exercise. Does the result match your fixed distribution?

Solution

from itertools import combinations

criteria = ["latency", "cost", "ops", "sdk", "community", "compliance"]

# For each pair: which is more important for internal RAG without an SRE?
my_preferences = {
    ("latency", "cost"): "cost",           # Internal: budget > speed
    ("latency", "ops"): "ops",             # No SRE: ops always wins
    ("latency", "sdk"): "latency",         # Latency > SDK
    ("latency", "community"): "latency",   # Latency > community
    ("latency", "compliance"): "latency",  # Latency > basic compliance
    ("cost", "ops"): "ops",                # No SRE: ops wins vs cost
    ("cost", "sdk"): "cost",               # Budget > SDK
    ("cost", "community"): "cost",         # Budget > community
    ("cost", "compliance"): "cost",        # Budget > compliance
    ("ops", "sdk"): "ops",                 # Ops > SDK
    ("ops", "community"): "ops",           # Ops > community
    ("ops", "compliance"): "ops",          # Ops > compliance
    ("sdk", "community"): "sdk",           # SDK > community
    ("sdk", "compliance"): "sdk",          # SDK > compliance
    ("community", "compliance"): "community",  # Community > compliance
}

weights = pairwise_comparison(criteria, my_preferences)
print("Pairwise weights:")
for c, w in sorted(weights.items(), key=lambda x: -x[1]):
    print(f"  {c}: {w:.1%}")

# Comparison: fixed distribution → ops 30%, cost 20%, latency 15%
# Pairwise will probably give: ops > cost > latency (consistent)

Exercise 3: Scoring with rubrics

Score ChromaDB and Pinecone using the rubrics from the previous section for these criteria: latency_p95, cost_monthly, ops_hours. Justify each score with evidence.

Solution

scoring_with_evidence = {
    "ChromaDB": {
        "latency_p95": {
            "score": 0.5,
            "raw_value": "300-500ms (self-hosted, depends on hardware)",
            "evidence": "No official p95 benchmarks. Estimate from community reports.",
            "rubric_level": "Partial compliance"
        },
        "cost_monthly": {
            "score": 1.0,
            "raw_value": "$0-60/month (self-hosted on a small VM)",
            "evidence": "Open source + VM cost. Effective free tier up to 100K.",
            "rubric_level": "Fully meets"
        },
        "ops_hours": {
            "score": 0.25,
            "raw_value": "10-15h/month (no mature managed option)",
            "evidence": "Requires manual monitoring, manual backups, updates.",
            "rubric_level": "Weak compliance"
        }
    },
    "Pinecone": {
        "latency_p95": {
            "score": 0.75,
            "raw_value": "~120ms (per public benchmarks)",
            "evidence": "Official benchmarks and ANN-benchmarks community.",
            "rubric_level": "Meets well"
        },
        "cost_monthly": {
            "score": 0.5,
            "raw_value": "$70-350/month (depends on tier and vectors)",
            "evidence": "Official pricing page. Free tier up to 100K 1536d vectors.",
            "rubric_level": "Partial compliance (for a budget < $200)"
        },
        "ops_hours": {
            "score": 1.0,
            "raw_value": "1-2h/month (fully managed)",
            "evidence": "No own infra required. Only integration monitoring.",
            "rubric_level": "Fully meets"
        }
    }
}

for provider, criteria in scoring_with_evidence.items():
    print(f"\n=== {provider} ===")
    for criterion, data in criteria.items():
        print(f"  {criterion}: {data['score']} ({data['rubric_level']})")
        print(f"    Raw: {data['raw_value']}")
        print(f"    Evidence: {data['evidence']}")

Exercise 4: Sensitivity analysis

Using the ScoringEngine, check whether your ranking changes when you vary the weight of "cost_monthly" from 1 to 5. Is the decision robust or fragile?

Solution

engine_test = ScoringEngine([
    CriterionConfig("latency_p95", weight=4.0, min_acceptable=500,
                    ideal_value=100, lower_is_better=True),
    CriterionConfig("cost_monthly", weight=3.0, min_acceptable=500,
                    ideal_value=100, lower_is_better=True),
    CriterionConfig("ops_hours", weight=5.0, min_acceptable=15,
                    ideal_value=2, lower_is_better=True),
])

engine_test.add_provider("Pinecone", {
    "latency_p95": 120, "cost_monthly": 350, "ops_hours": 2
})
engine_test.add_provider("Qdrant Cloud", {
    "latency_p95": 150, "cost_monthly": 200, "ops_hours": 3
})
engine_test.add_provider("ChromaDB Self", {
    "latency_p95": 350, "cost_monthly": 60, "ops_hours": 12
})

results = engine_test.sensitivity_analysis("cost_monthly", [1.0, 2.0, 3.0, 4.0, 5.0])

print("Sensitivity to the weight of 'cost_monthly':")
for weight, ranking in results.items():
    leader = ranking[0]
    print(f"  Weight {weight:.0f}: Winner = {leader[0]} ({leader[1]}%)")

# If the winner changes from Pinecone to ChromaDB when cost rises to 5,
# your decision is sensitive to the cost criterion. Document this.

Exercise 5: Bias detection

Review this hypothetical scoring and detect which biases might be present:

Criterion	"Our current provider"	"New provider"
Latency	0.9	0.7
Cost	0.8	0.8
Ops	0.9	0.6
SDK	0.95	0.5
Community	0.85	0.4

Solution

analysis = {
    "familiarity_bias": {
        "detected": True,
        "evidence": "The current provider has scores > 0.8 on EVERYTHING. "
                    "It's statistically unlikely it's superior in every dimension.",
        "action": "Include an evaluator who has NOT used the current provider"
    },
    "halo_effect": {
        "detected": True,
        "evidence": "SDK 0.95 is suspiciously high. "
                    "Is the SDK really 'almost perfect'? Or have you already learned its quirks?",
        "action": "Apply an objective rubric to the SDK: type hints, async, docs, testing mode"
    },
    "anchoring_bias": {
        "detected": True,
        "evidence": "If you evaluated the current one first, the new one's scores are compared "
                    "against the current one instead of against the rubric",
        "action": "Evaluate both against the rubric, not one against the other"
    },
    "sunk_cost": {
        "detected": "Possible",
        "evidence": "Community 0.85 vs 0.4 is a huge gap. "
                    "Does the current one really have a better community, or do YOU just know its community more?",
        "action": "Measure community with objective metrics: stars, SO answers, integration count"
    }
}

for bias, info in analysis.items():
    status = "🔴" if info["detected"] == True else "🟡"
    print(f"{status} {bias}: {info['evidence']}")
    print(f"   → {info['action']}")
    print()

🔗 Connection with the project: Decision Questionnaire

In your Decision Questionnaire, the ScoringEngine from this capsule is the central calculation engine:

The questionnaire collects inputs → they become CriterionConfig (weights and thresholds)
The provider data → is fed in as raw_scores
The engine calculates → normalization + weighting
The final report includes ranking, gap analysis, and a sensitivity check

Reuse the ScoringEngine, normalize_score, and sensitivity_analysis code directly in your project.

Summary

Weights determine the result more than the scores. Invest time in the weights.
Three techniques for assigning weights: fixed distribution (forces trade-offs), pairwise comparison (intuitive), stack ranking (fast).
Without a rubric, the score is opinion. Define what 0.5 vs 0.75 means for each criterion.
Normalize before weighting. Latency in ms and cost in USD can't be added directly.
Do a sensitivity analysis. If changing a weight by ±1 changes the winner, your decision is fragile.
Detect biases systematically. Anchoring, familiarity, recency, and halo effect are the most common.
Ties aren't resolved with more decimals. If two options are within ±3%, decide by operational simplicity.

Additional resources

Weighted Sum Model (Wikipedia) — Theoretical foundation of the weighted scoring matrix
Pairwise Comparison Method — Pairwise comparison method for prioritization
MoSCoW Method — A complementary prioritization framework
AHP (Analytic Hierarchy Process) — Advanced multi-criteria decision method
Cognitive Biases in Decision Making — List of relevant cognitive biases
ANN Benchmarks — Objective data for performance scoring
MCDA (Multi-Criteria Decision Analysis) — Academic framework for complex decisions
Decision Quality Framework (SDG) — Professional decision quality framework

Estimated time: 25-35 minutes Next: 04-practical-decision-matrix.md

← PreviousCapsule 02: Requirements and Decision Criteria Next →Capsule 04: Practical Decision Matrix

Capsule 03: Weights, Scoring, and Matrix Methodology

🎯 Capsule objective

Capsule description

Why weights matter more than scores

Technique 1: Fixed point distribution

Technique 2: Pairwise comparison

Technique 3: Stack ranking

Scoring scales: concrete rubrics

5-level scale (recommended)

Dimension-specific rubrics

Normalization: comparing different dimensions

ScoringEngine: complete implementation

Interpreting results

Sensitivity analysis

Bias in scoring: how to detect and reduce it

🔧 Troubleshooting

Problem 1: "Everything comes out tied"

Problem 2: "The result contradicts the team's intuition"

Problem 3: "I don't have data to score objectively"

Problem 4: "Each stakeholder produces a completely different ranking"

Problem 5: "A provider wins because of the weights, not because it's better"

🏋️ Exercises

Exercise 1: Fixed point distribution

Exercise 2: Pairwise comparison

Exercise 3: Scoring with rubrics

Exercise 4: Sensitivity analysis

Exercise 5: Bias detection

🔗 Connection with the project: Decision Questionnaire

Summary

Additional resources