Module 6: Decision Matrix for AI Engineers
Capsule 04: Practical Decision Matrix
🎯 Capsule objective
Apply the entire framework (requirements, weights, scoring, normalization) to a full real case: evaluate 5 vector databases for a concrete RAG project, from defining the criteria to the final documented recommendation.
By the end of this capsule:
- ✅ You'll run a complete evaluation of 5 providers on a real case
- ✅ You'll build a reproducible decision matrix in Python
- ✅ You'll generate a report with ranking, gap analysis, and recommendation
- ✅ You'll document trade-offs and a re-evaluation plan
Estimated time: 30-40 minutes
Capsule description
You've built the pieces: criteria with thresholds (capsule 02), weights with methodology (capsule 03). Now it's time to assemble everything into a complete hands-on exercise. In this capsule you're going to evaluate 5 real vector databases — Pinecone, Qdrant Cloud, Weaviate Cloud, Milvus, and ChromaDB — for a concrete use case: a RAG chatbot for the technical documentation of a growing startup.
The value of this capsule isn't only in the result (which one "wins"), but in the process. You'll see how the same methodology produces different results depending on the team's context, and how to document the decision so that your "future self" understands why you chose what you chose.
When you finish, you'll have an evaluation template you can reuse every time you need to evaluate a new provider or re-evaluate your current stack.
The case: RAG Chatbot for technical documentation
Project context
project_context = {
"name": "DocBot — RAG for technical docs of a SaaS product",
"description": "Chatbot that answers user questions about the documentation "
"of a B2B SaaS product (API docs, guides, troubleshooting)",
"team": {
"size": 5,
"roles": ["2 backend (Python)", "1 frontend", "1 ML engineer", "1 PM"],
"devops": "No dedicated role — backend handles infra as a side-task",
"vector_db_experience": "Used ChromaDB in a PoC 3 months ago"
},
"scale": {
"current_docs": 2_500,
"current_vectors": 75_000, # ~30 chunks per doc
"projected_docs_12m": 15_000,
"projected_vectors_12m": 450_000,
"daily_queries": 500,
"projected_daily_queries_12m": 5_000
},
"requirements": {
"latency_target": "p95 < 300ms (streaming chatbot, first chunk matters)",
"budget": "$400/month max for vector DB (total infra budget $2K/month)",
"compliance": "SOC 2 on roadmap for Q3, not mandatory yet",
"features": "Metadata filtering mandatory, hybrid search desirable",
"integrations": "LangChain, OpenAI embeddings (text-embedding-3-small, 1536d)"
},
"constraints": {
"timeline": "Production in 6 weeks",
"migration_tolerance": "Low — if they pick wrong, migrating costs ~2 sprints",
"risk_tolerance": "Medium — can tolerate 1-2 minor incidents/month"
}
}
print(f"Project: {project_context['name']}")
print(f"Team: {project_context['team']['size']} people, DevOps: {project_context['team']['devops']}")
print(f"Vectors: {project_context['scale']['current_vectors']:,} → "
f"{project_context['scale']['projected_vectors_12m']:,} (12m)")
print(f"Budget: {project_context['requirements']['budget']}")Step 1: Define criteria and weights
Based on the project context, define the relevant criteria:
from dataclasses import dataclass, field
from enum import Enum
class Priority(Enum):
ELIMINATORY = "eliminatory"
CRITICAL = "critical"
IMPORTANT = "important"
NICE_TO_HAVE = "nice_to_have"
@dataclass
class CriterionConfig:
name: str
weight: float
min_acceptable: float
ideal_value: float
lower_is_better: bool = True
unit: str = ""
priority: Priority = Priority.IMPORTANT
rationale: str = ""
criteria_config = [
CriterionConfig(
name="latency_p95",
weight=4.0,
min_acceptable=500,
ideal_value=150,
lower_is_better=True,
unit="ms",
priority=Priority.CRITICAL,
rationale="Chatbot: first chunk must arrive fast for streaming UX"
),
CriterionConfig(
name="scale_vectors",
weight=4.0,
min_acceptable=500_000,
ideal_value=5_000_000,
lower_is_better=False,
unit="vectors",
priority=Priority.CRITICAL,
rationale="450K at 12 months + margin. We don't want to migrate in 18 months"
),
CriterionConfig(
name="ops_simplicity",
weight=5.0,
min_acceptable=15,
ideal_value=3,
lower_is_better=True,
unit="hours/month",
priority=Priority.CRITICAL,
rationale="No dedicated DevOps. Every ops hour = one less product hour"
),
CriterionConfig(
name="monthly_cost",
weight=4.0,
min_acceptable=400,
ideal_value=100,
lower_is_better=True,
unit="USD",
priority=Priority.CRITICAL,
rationale="$400/month budget. It can't consume the whole infra budget"
),
CriterionConfig(
name="sdk_quality",
weight=3.0,
min_acceptable=0.5,
ideal_value=0.9,
lower_is_better=False,
unit="score 0-1",
priority=Priority.IMPORTANT,
rationale="5 devs, 2 backend Python. A bad SDK = weeks lost"
),
CriterionConfig(
name="metadata_filtering",
weight=3.0,
min_acceptable=0.6,
ideal_value=1.0,
lower_is_better=False,
unit="score 0-1",
priority=Priority.CRITICAL,
rationale="Filtering by product, version, doc type is mandatory"
),
CriterionConfig(
name="hybrid_search",
weight=2.0,
min_acceptable=0.0,
ideal_value=1.0,
lower_is_better=False,
unit="score 0-1",
priority=Priority.NICE_TO_HAVE,
rationale="Desirable for queries with exact API names"
),
CriterionConfig(
name="community_ecosystem",
weight=2.0,
min_acceptable=0.4,
ideal_value=0.9,
lower_is_better=False,
unit="score 0-1",
priority=Priority.IMPORTANT,
rationale="LangChain integration + answers on SO when we hit issues"
),
]
print("=== Defined criteria ===")
print(f"{'Criterion':<25} {'Weight':<6} {'Priority':<15} {'Direction':<15}")
print("-" * 65)
for c in criteria_config:
direction = "↓ lower better" if c.lower_is_better else "↑ higher better"
print(f"{c.name:<25} {c.weight:<6.1f} {c.priority.value:<15} {direction}")
total_weight = sum(c.weight for c in criteria_config)
print(f"\nTotal weight: {total_weight}")Step 2: Collect provider data
The raw scores are based on official documentation, public benchmarks, and PoC experience:
provider_data = {
"Pinecone": {
"latency_p95": 120,
"scale_vectors": 10_000_000,
"ops_simplicity": 2, # h/month — fully managed
"monthly_cost": 350, # Starter tier for 450K vectors 1536d
"sdk_quality": 0.85,
"metadata_filtering": 0.95, # Native, robust, full operators
"hybrid_search": 0.9, # Native sparse-dense
"community_ecosystem": 0.9, # LangChain, LlamaIndex, broad community
"notes": "Pure managed. Excellent DX. Cost scales with vectors.",
"evidence": {
"latency": "ANN-benchmarks + official docs",
"cost": "Pricing page, calculator for 450K 1536d",
"ops": "Fully managed, no infra required",
"sdk": "Tested in internal PoC + docs review"
}
},
"Qdrant Cloud": {
"latency_p95": 140,
"scale_vectors": 50_000_000,
"ops_simplicity": 3, # h/month — managed with config
"monthly_cost": 180, # Cloud tier for 450K vectors
"sdk_quality": 0.80,
"metadata_filtering": 0.90, # Robust payload filtering
"hybrid_search": 0.85, # Sparse vectors + fusion
"community_ecosystem": 0.75, # Growing fast, LangChain ok
"notes": "Good managed/control balance. Rust engine, fast.",
"evidence": {
"latency": "ANN-benchmarks, community reports",
"cost": "Qdrant Cloud pricing page",
"ops": "Dashboard + API, minimal config",
"sdk": "qdrant-client Python review"
}
},
"Weaviate Cloud": {
"latency_p95": 180,
"scale_vectors": 20_000_000,
"ops_simplicity": 3,
"monthly_cost": 280,
"sdk_quality": 0.75,
"metadata_filtering": 0.85,
"hybrid_search": 0.95, # Native BM25 + vector, best hybrid
"community_ecosystem": 0.80,
"notes": "Best native hybrid search. Built-in vectorization modules.",
"evidence": {
"latency": "Weaviate benchmarks blog + community",
"cost": "Weaviate pricing, serverless tier",
"ops": "WCD dashboard, auto-scaling",
"sdk": "weaviate-client Python v4 review"
}
},
"Milvus (Zilliz Cloud)": {
"latency_p95": 100,
"scale_vectors": 100_000_000,
"ops_simplicity": 5, # Zilliz Cloud managed
"monthly_cost": 220,
"sdk_quality": 0.70,
"metadata_filtering": 0.80,
"hybrid_search": 0.75,
"community_ecosystem": 0.70,
"notes": "Maximum scale. Python SDK less polished than competitors.",
"evidence": {
"latency": "ANN-benchmarks (top performer)",
"cost": "Zilliz Cloud pricing",
"ops": "Zilliz managed, moderate configuration",
"sdk": "pymilvus review, docs review"
}
},
"ChromaDB": {
"latency_p95": 350,
"scale_vectors": 500_000,
"ops_simplicity": 12, # Self-hosted, no mature managed option
"monthly_cost": 60, # Only VM cost
"sdk_quality": 0.88,
"metadata_filtering": 0.75, # Basic, limited operators
"hybrid_search": 0.0, # No native hybrid support
"community_ecosystem": 0.85, # LangChain first-class, large community
"notes": "Best DX for prototyping. Limited scale. No hybrid.",
"evidence": {
"latency": "Internal PoC (3 months ago)",
"cost": "Open source + t3.medium AWS",
"ops": "Own experience: manual backups, custom monitoring",
"sdk": "Team's direct experience"
}
}
}
print(f"Providers to evaluate: {len(provider_data)}")
for name, data in provider_data.items():
print(f"\n {name}:")
print(f" {data['notes']}")Step 3: Normalization and scoring
def normalize_score(raw_value, min_acceptable, ideal_value, lower_is_better=True):
"""Normalize a raw value to a 0-1 scale."""
if lower_is_better:
if raw_value <= ideal_value:
return 1.0
elif raw_value >= min_acceptable:
return 0.25
else:
range_size = min_acceptable - ideal_value
if range_size == 0:
return 1.0
return max(0.0, min(1.0, round(1.0 - ((raw_value - ideal_value) / range_size), 2)))
else:
if raw_value >= ideal_value:
return 1.0
elif raw_value <= min_acceptable:
return 0.25
else:
range_size = ideal_value - min_acceptable
if range_size == 0:
return 1.0
return max(0.0, min(1.0, round((raw_value - min_acceptable) / range_size, 2)))
def run_evaluation(criteria: list, providers: dict) -> dict:
"""Run the full evaluation."""
results = {}
for provider_name, raw_data in providers.items():
scores = {}
weighted_scores = {}
total_weighted = 0
max_weighted = 0
for criterion in criteria:
raw = raw_data.get(criterion.name, 0)
normalized = normalize_score(
raw, criterion.min_acceptable,
criterion.ideal_value, criterion.lower_is_better
)
weighted = normalized * criterion.weight
scores[criterion.name] = {
"raw": raw,
"normalized": normalized,
"weighted": round(weighted, 2)
}
total_weighted += weighted
max_weighted += criterion.weight
percentage = round((total_weighted / max_weighted) * 100, 1)
results[provider_name] = {
"scores": scores,
"total_weighted": round(total_weighted, 2),
"max_possible": max_weighted,
"percentage": percentage
}
return results
results = run_evaluation(criteria_config, provider_data)
# Show results table
print("\n" + "=" * 100)
print("DECISION MATRIX — DocBot RAG Chatbot")
print("=" * 100)
header = f"{'Criterion':<25} {'Weight':<5}"
for name in provider_data:
header += f" {name:<16}"
print(header)
print("-" * 100)
for criterion in criteria_config:
row = f"{criterion.name:<25} {criterion.weight:<5.1f}"
for provider_name in provider_data:
data = results[provider_name]["scores"][criterion.name]
row += f" {data['raw']:>6} → {data['normalized']:.2f} "
print(row)
print("-" * 100)
total_row = f"{'FINAL SCORE':<25} {'':5}"
for provider_name in provider_data:
pct = results[provider_name]["percentage"]
total_row += f" {pct:>12.1f}% "
print(total_row)
print("=" * 100)Step 4: Ranking and analysis
def generate_ranking(results: dict) -> list[tuple]:
"""Generate an ordered ranking."""
ranking = [
(name, data["percentage"])
for name, data in results.items()
]
return sorted(ranking, key=lambda x: -x[1])
def gap_analysis(results: dict, criteria: list, provider_name: str) -> list[dict]:
"""Identify where a provider loses the most points."""
provider = results[provider_name]
gaps = []
for criterion in criteria:
score_data = provider["scores"][criterion.name]
if score_data["normalized"] < 0.75:
lost = (1.0 - score_data["normalized"]) * criterion.weight
gaps.append({
"criterion": criterion.name,
"normalized": score_data["normalized"],
"raw": score_data["raw"],
"weight": criterion.weight,
"points_lost": round(lost, 2),
"unit": criterion.unit
})
return sorted(gaps, key=lambda g: -g["points_lost"])
ranking = generate_ranking(results)
print("\n=== FINAL RANKING ===\n")
for i, (name, pct) in enumerate(ranking, 1):
if pct >= 80:
verdict = "✅ Excellent fit"
elif pct >= 65:
verdict = "🟡 Good fit with trade-offs"
elif pct >= 50:
verdict = "🟠 Partial fit"
else:
verdict = "🔴 Not recommended"
print(f" #{i} {name:<25} {pct:>6.1f}% {verdict}")
# Gap analysis for the top 2
print("\n=== GAP ANALYSIS ===")
for name, _ in ranking[:2]:
gaps = gap_analysis(results, criteria_config, name)
print(f"\n {name} — Areas to improve:")
if not gaps:
print(" No significant gaps")
for gap in gaps:
print(f" ⚠️ {gap['criterion']}: score {gap['normalized']:.2f} "
f"(raw: {gap['raw']} {gap['unit']}) — loses {gap['points_lost']} pts")Step 5: Head-to-head comparison of the top 2
def head_to_head(results: dict, criteria: list, provider_a: str, provider_b: str):
"""Direct comparison between two providers."""
print(f"\n{'=' * 70}")
print(f"HEAD TO HEAD: {provider_a} vs {provider_b}")
print(f"{'=' * 70}\n")
a_wins = 0
b_wins = 0
ties = 0
print(f"{'Criterion':<25} {'Weight':<5} {provider_a:<15} {provider_b:<15} {'Winner':<15}")
print("-" * 75)
for criterion in criteria:
a_score = results[provider_a]["scores"][criterion.name]["normalized"]
b_score = results[provider_b]["scores"][criterion.name]["normalized"]
if a_score > b_score:
winner = provider_a
a_wins += 1
elif b_score > a_score:
winner = provider_b
b_wins += 1
else:
winner = "Tie"
ties += 1
print(f"{criterion.name:<25} {criterion.weight:<5.1f} "
f"{a_score:<15.2f} {b_score:<15.2f} {winner}")
print("-" * 75)
a_pct = results[provider_a]["percentage"]
b_pct = results[provider_b]["percentage"]
print(f"{'TOTAL':<25} {'':5} {a_pct:<15.1f} {b_pct:<15.1f}")
print(f"\nWins by criterion: {provider_a}={a_wins}, {provider_b}={b_wins}, Ties={ties}")
gap = abs(a_pct - b_pct)
if gap < 3:
print(f"⚠️ Difference of {gap:.1f}% — FRAGILE decision. Consider a tie-breaker criterion.")
elif gap < 8:
print(f"🟡 Difference of {gap:.1f}% — moderate advantage.")
else:
print(f"✅ Difference of {gap:.1f}% — clear advantage.")
# Run head-to-head for the top 2
top_2 = [name for name, _ in ranking[:2]]
head_to_head(results, criteria_config, top_2[0], top_2[1])Step 6: Sensitivity analysis
def sensitivity_analysis(criteria: list, providers: dict, criterion_name: str,
weight_range: list[float]) -> dict:
"""Does the winner change if I adjust one criterion's weight?"""
original_weight = None
target = None
for c in criteria:
if c.name == criterion_name:
original_weight = c.weight
target = c
break
analysis = {}
for new_weight in weight_range:
target.weight = new_weight
temp_results = run_evaluation(criteria, providers)
temp_ranking = generate_ranking(temp_results)
analysis[new_weight] = temp_ranking
target.weight = original_weight
return analysis
print("\n=== SENSITIVITY ANALYSIS ===\n")
sensitive_criteria = ["ops_simplicity", "monthly_cost", "latency_p95"]
for criterion_name in sensitive_criteria:
analysis = sensitivity_analysis(
criteria_config, provider_data,
criterion_name, [1.0, 2.0, 3.0, 4.0, 5.0]
)
print(f"\nSensitivity: '{criterion_name}'")
for weight, ranking_list in analysis.items():
leader = ranking_list[0]
second = ranking_list[1]
gap = leader[1] - second[1]
stability = "🟢" if gap > 5 else "🟡" if gap > 2 else "🔴"
print(f" Weight {weight:.0f}: #{1} {leader[0]:<20} ({leader[1]:.1f}%) "
f"vs #{2} {second[0]:<20} ({second[1]:.1f}%) {stability}")Step 7: Final documented recommendation
def generate_recommendation(ranking: list, results: dict, criteria: list,
project_context: dict) -> str:
"""Generate a formal recommendation document."""
winner = ranking[0]
alternative = ranking[1]
lines = [
"=" * 70,
"VECTOR DATABASE RECOMMENDATION",
f"Project: {project_context['name']}",
f"Date: 2025-03 (re-evaluate: 2025-09)",
"=" * 70,
"",
f"PRIMARY RECOMMENDATION: {winner[0]} ({winner[1]:.1f}%)",
f"ALTERNATIVE: {alternative[0]} ({alternative[1]:.1f}%)",
"",
"--- RATIONALE ---",
]
winner_gaps = gap_analysis(results, criteria, winner[0])
alt_gaps = gap_analysis(results, criteria, alternative[0])
lines.append(f"\n{winner[0]} wins because:")
winner_strengths = [
c.name for c in criteria
if results[winner[0]]["scores"][c.name]["normalized"] >= 0.75
]
for s in winner_strengths[:4]:
score = results[winner[0]]["scores"][s]["normalized"]
lines.append(f" ✅ {s}: {score:.2f}")
if winner_gaps:
lines.append(f"\nRisks of {winner[0]}:")
for gap in winner_gaps[:3]:
lines.append(f" ⚠️ {gap['criterion']}: score {gap['normalized']:.2f} "
f"(raw: {gap['raw']} {gap['unit']})")
lines.extend([
"",
"--- ACTION PLAN ---",
"1. 3-day PoC with the recommended provider",
"2. Benchmark with a real dataset (1000 project docs)",
"3. Validate p95 latency with simulated load",
"4. Review pricing for 450K vectors (12m projection)",
"5. Final decision in sprint planning",
"",
"--- RE-EVALUATION TRIGGERS ---",
f"• Vectors exceed {project_context['scale']['projected_vectors_12m']:,}",
f"• Monthly cost exceeds ${project_context['requirements']['budget'].split('$')[1].split('/')[0]}",
"• SOC 2 becomes mandatory (Q3 roadmap)",
"• p95 latency > 400ms sustained for 1 week",
"• A disruptive new provider enters the market",
])
return "\n".join(lines)
recommendation = generate_recommendation(ranking, results, criteria_config, project_context)
print(recommendation)Full step: all-in-one function
Here's a function that runs the whole process from start to finish:
def full_evaluation_pipeline(project_name: str, criteria: list,
providers: dict) -> dict:
"""
Complete evaluation pipeline.
Returns results, ranking, gaps, and recommendation.
"""
# 1. Evaluation
results = run_evaluation(criteria, providers)
# 2. Ranking
ranking = generate_ranking(results)
# 3. Gap analysis for the top 3
gaps = {}
for name, _ in ranking[:3]:
gaps[name] = gap_analysis(results, criteria, name)
# 4. Head-to-head of the top 2
top_2 = [name for name, _ in ranking[:2]]
# 5. Summary
summary = {
"project": project_name,
"providers_evaluated": len(providers),
"criteria_count": len(criteria),
"total_weight": sum(c.weight for c in criteria),
"ranking": ranking,
"recommendation": ranking[0][0],
"alternative": ranking[1][0],
"gap_top_2": abs(ranking[0][1] - ranking[1][1]),
"decision_stability": "robust" if abs(ranking[0][1] - ranking[1][1]) > 5 else "fragile",
"gaps": gaps
}
return summary
pipeline_result = full_evaluation_pipeline(
"DocBot RAG", criteria_config, provider_data
)
print(f"\nPipeline summary:")
print(f" Recommendation: {pipeline_result['recommendation']}")
print(f" Alternative: {pipeline_result['alternative']}")
print(f" Gap: {pipeline_result['gap_top_2']:.1f}%")
print(f" Stability: {pipeline_result['decision_stability']}")Reusable template
Every time you need to evaluate a new provider or re-evaluate your stack:
EVALUATION_TEMPLATE = """
# Vector Database Evaluation
# Date: {date}
# Project: {project}
# Evaluator: {evaluator}
## 1. Context
- Team: {team_size} people, DevOps: {devops}
- Vectors: {current_vectors:,} → {projected_vectors:,} (12m)
- Budget: ${budget}/month
- Timeline: {timeline}
## 2. Criteria (from the Requirements Document)
{criteria_table}
## 3. Providers evaluated
{providers_list}
## 4. Results
{results_table}
## 5. Recommendation
- Primary: {recommendation}
- Alternative: {alternative}
- Rationale: {justification}
## 6. Re-evaluation triggers
{triggers}
## 7. Sign-off
- Decision made by: {decision_makers}
- Scheduled re-evaluation date: {review_date}
"""
def generate_template(context: dict) -> str:
"""Generate the evaluation document from the template."""
return EVALUATION_TEMPLATE.format(**context)🔧 Troubleshooting
Problem 1: "The matrix becomes too complex"
Symptom: More than 10 criteria, 6+ providers, the table doesn't fit on any screen.
Solution: Apply phased elimination. First filter with eliminatory criteria (reduces providers). Then limit to 6 weighted criteria. If you still have 5+ providers, make a first cut with 3 main criteria and then evaluate the top 3 with the full matrix.
Problem 2: "We don't know how to score objectively"
Symptom: The scores are based on "I think..." instead of evidence.
Solution: Define the evidence source for each score: public benchmark (ANN-benchmarks), official documentation (pricing, features), your own PoC (real measurements). Mark each score with its source. If there's no source → provisional score with ⚠️.
Problem 3: "The business changed mid-analysis"
Symptom: You started with "MVP without compliance" and now they're asking for SOC 2.
Solution: Don't redo the whole evaluation. Update only the affected weights and thresholds. If the change adds an eliminatory criterion (like SOC 2), re-filter providers first and then recalculate scores only for the ones that pass.
Problem 4: "Two providers are technically tied"
Symptom: Difference < 3% between #1 and #2.
Solution: A technical tie isn't resolved with more decimals. Use tie-breaker criteria in this order: (1) operational simplicity, (2) the team's prior experience, (3) setup speed for a PoC. If everything is still tied, choose the one with the lowest future migration cost.
Problem 5: "A stakeholder wants to override the matrix"
Symptom: The CTO says "let's use X because I know it" ignoring the scores.
Solution: The override is valid but must be documented. Create an "Override justification" section in the decision document explaining why experience/relationship/strategic fit outweighs the scoring. This protects the team if the decision turns out to be wrong.
🏋️ Exercises
Exercise 1: Evaluation for a different scenario
Change the context: now you're an enterprise company with 50 developers, dedicated DevOps, 5M current vectors, a $5K/month budget, and mandatory SOC 2. How do the weights change? Does the winner change?
Solution
enterprise_criteria = [
CriterionConfig("latency_p95", weight=5.0, min_acceptable=300,
ideal_value=50, lower_is_better=True, unit="ms"),
CriterionConfig("scale_vectors", weight=5.0, min_acceptable=10_000_000,
ideal_value=100_000_000, lower_is_better=False, unit="vectors"),
CriterionConfig("ops_simplicity", weight=2.0, min_acceptable=25,
ideal_value=5, lower_is_better=True, unit="hours/month"),
CriterionConfig("monthly_cost", weight=3.0, min_acceptable=5000,
ideal_value=1000, lower_is_better=True, unit="USD"),
CriterionConfig("sdk_quality", weight=2.0, min_acceptable=0.5,
ideal_value=0.9, lower_is_better=False, unit="score"),
CriterionConfig("metadata_filtering", weight=4.0, min_acceptable=0.7,
ideal_value=1.0, lower_is_better=False, unit="score"),
CriterionConfig("hybrid_search", weight=3.0, min_acceptable=0.5,
ideal_value=1.0, lower_is_better=False, unit="score"),
CriterionConfig("community_ecosystem", weight=2.0, min_acceptable=0.5,
ideal_value=0.9, lower_is_better=False, unit="score"),
]
enterprise_results = run_evaluation(enterprise_criteria, provider_data)
enterprise_ranking = generate_ranking(enterprise_results)
print("=== Enterprise Ranking ===")
for i, (name, pct) in enumerate(enterprise_ranking, 1):
print(f" #{i} {name}: {pct:.1f}%")
# With enterprise: scale dominates → Milvus/Qdrant rise
# ops_simplicity drops → self-hosted ChromaDB is no longer penalized as much
# hybrid_search rises → Weaviate benefitsExercise 2: Add a new provider
pgvector (PostgreSQL with the vector extension) just came onto your radar. Research it and add its raw scores to the evaluation. Where does it land in the ranking?
Solution
provider_data["pgvector"] = {
"latency_p95": 250, # Depends on PostgreSQL tuning
"scale_vectors": 2_000_000, # Limited by RAM/configuration
"ops_simplicity": 8, # If you already have PostgreSQL, less overhead
"monthly_cost": 80, # Only the PostgreSQL cost (you already have it)
"sdk_quality": 0.65, # SQLAlchemy + pgvector, no dedicated SDK
"metadata_filtering": 0.70, # SQL WHERE — powerful but different
"hybrid_search": 0.6, # pg_trgm + vector, manual but functional
"community_ecosystem": 0.60, # PostgreSQL huge, pgvector-specific is smaller
"notes": "If you already have PostgreSQL, it's the lowest-friction option. "
"It's not a dedicated vector DB, but enough for many cases.",
"evidence": {
"latency": "Community benchmarks pgvector vs dedicated",
"cost": "You already pay for PostgreSQL — minimal incremental cost",
"ops": "Your DBA already operates PostgreSQL",
"sdk": "No dedicated SDK — you use SQL + the extension"
}
}
updated_results = run_evaluation(criteria_config, provider_data)
updated_ranking = generate_ranking(updated_results)
print("=== Ranking with pgvector ===")
for i, (name, pct) in enumerate(updated_ranking, 1):
print(f" #{i} {name}: {pct:.1f}%")
# pgvector typically lands in the middle: low cost but
# ops/latency/features don't compete with dedicated vector DBs
# Except if you already have PostgreSQL in production and want to minimize the stack
del provider_data["pgvector"] # clean up for the next exercisesExercise 3: Simulating an eliminatory-criterion failure
SOC 2 just became mandatory (it was nice-to-have before). Add an eliminatory criterion and re-evaluate. How many providers survive?
Solution
soc2_compliance = {
"Pinecone": True, # SOC 2 Type II certified
"Qdrant Cloud": True, # SOC 2 certified
"Weaviate Cloud": True, # SOC 2 in progress / available
"Milvus (Zilliz Cloud)": True, # Zilliz: SOC 2 certified
"ChromaDB": False, # Open source, no certifications
}
print("=== Eliminatory filter: SOC 2 ===\n")
surviving = []
for provider, compliant in soc2_compliance.items():
status = "✅ PASS" if compliant else "❌ ELIMINATED"
print(f" {provider}: {status}")
if compliant:
surviving.append(provider)
print(f"\nSurvivors: {len(surviving)}/{len(soc2_compliance)}")
print(f"Eliminated: ChromaDB (open source without SOC 2)")
# Re-evaluate only the survivors
filtered_providers = {k: v for k, v in provider_data.items() if k in surviving}
filtered_results = run_evaluation(criteria_config, filtered_providers)
filtered_ranking = generate_ranking(filtered_results)
print("\n=== Post-elimination ranking ===")
for i, (name, pct) in enumerate(filtered_ranking, 1):
print(f" #{i} {name}: {pct:.1f}%")Exercise 4: Complete decision document
Generate the full decision document for the DocBot case using the template. Include context, criteria, results, recommendation, and re-evaluation triggers.
Solution
decision_document = {
"date": "2025-03-15",
"project": "DocBot — RAG Chatbot for technical docs",
"evaluator": "Backend Team",
"team_size": 5,
"devops": "No dedicated role",
"current_vectors": 75_000,
"projected_vectors": 450_000,
"budget": 400,
"timeline": "6 weeks to production",
"criteria_table": "\n".join([
f"| {c.name} | {c.weight} | {c.priority.value} | {c.unit} |"
for c in criteria_config
]),
"providers_list": ", ".join(provider_data.keys()),
"results_table": "\n".join([
f"| #{i} | {name} | {pct:.1f}% |"
for i, (name, pct) in enumerate(ranking, 1)
]),
"recommendation": ranking[0][0],
"alternative": ranking[1][0],
"justification": (
f"{ranking[0][0]} wins with {ranking[0][1]:.1f}% vs "
f"{ranking[1][0]} with {ranking[1][1]:.1f}%. "
f"Difference of {abs(ranking[0][1] - ranking[1][1]):.1f}%. "
f"{'Robust decision.' if abs(ranking[0][1] - ranking[1][1]) > 5 else 'Fragile decision — validate with a PoC.'}"
),
"triggers": (
"- Vectors > 500K\n"
"- Cost > $400/month\n"
"- SOC 2 mandatory\n"
"- p95 > 400ms sustained\n"
"- New major release of the alternative provider"
),
"decision_makers": "CTO + Backend Lead",
"review_date": "2025-09-15 (6 months)"
}
doc = generate_template(decision_document)
print(doc)Exercise 5: Quarterly re-evaluation
Three months have passed since the decision. Vectors grew to 200K, the cost rose to $280/month, and the team reports 4h/month of maintenance. Is the original decision still valid? Re-run the evaluation with updated data.
Solution
# Data updated at 3 months
updated_context = {
"vectors_now": 200_000,
"vectors_projected_9m": 350_000, # projection adjustment
"actual_cost": 280,
"actual_ops_hours": 4,
"actual_latency_p95": 160,
"issues_reported": 1, # one minor incident
}
print("=== Quarterly Re-evaluation ===\n")
# Did any trigger fire?
triggers_check = {
"Vectors > 500K": updated_context["vectors_now"] > 500_000,
"Cost > $400/month": updated_context["actual_cost"] > 400,
"SOC 2 mandatory": False, # not yet
"p95 > 400ms": updated_context["actual_latency_p95"] > 400,
}
print("Re-evaluation triggers:")
any_triggered = False
for trigger, activated in triggers_check.items():
status = "🔴 FIRED" if activated else "🟢 OK"
print(f" {trigger}: {status}")
if activated:
any_triggered = True
if not any_triggered:
print("\n✅ No trigger fired. The original decision is still valid.")
print("Next scheduled re-evaluation: +3 months")
else:
print("\n⚠️ Trigger fired. Start the re-evaluation process.")
# Compare expected vs actual metrics
print("\n=== Metrics: expected vs actual ===")
comparisons = [
("p95 latency", "< 300ms", f"{updated_context['actual_latency_p95']}ms", "✅"),
("Monthly cost", "< $400/month", f"${updated_context['actual_cost']}/month", "✅"),
("Ops hours", "< 5h/month", f"{updated_context['actual_ops_hours']}h/month", "✅"),
("Incidents", "< 2/month", f"{updated_context['issues_reported']}/month", "✅"),
]
for metric, expected, actual, status in comparisons:
print(f" {metric}: expected {expected}, actual {actual} {status}")🔗 Project connection: Decision Questionnaire
Your Decision Questionnaire will implement exactly this pipeline. The difference is that instead of hardcoding the values, your questionnaire:
- Collects the project context interactively (team, budget, vectors)
- Generates the criteria based on the answers (using the logic from capsule 02)
- Assigns weights guiding the user through the fixed-distribution technique
- Runs the scoring with pre-loaded provider data
- Generates the report with ranking, gaps, and recommendation
The full_evaluation_pipeline() from this capsule is the core of your project.
Summary
- The complete process has 7 steps: context → criteria → weights → data → scoring → analysis → recommendation.
- Provider data needs sources. Benchmark, docs, PoC. Never marketing.
- Gap analysis reveals where the risks are in your winning option.
- Head-to-head of the top 2 is mandatory: if they're < 3% apart, the decision is fragile.
- Sensitivity analysis tells you whether your decision depends on a single weight.
- The recommendation includes re-evaluation triggers. It's not "forever".
- Document EVERYTHING: criteria, weights, scores, sources, rationale, override if it applies.
- Re-evaluate quarterly or when a trigger fires.
Additional resources
- ANN Benchmarks — Objective performance benchmarks for scoring
- Pinecone Pricing Calculator — Cost estimation for cost scoring
- Qdrant Cloud Pricing — Pricing data for evaluation
- Weaviate Pricing — Weaviate pricing and tiers
- Zilliz Cloud (Milvus) — Managed Milvus pricing
- pgvector GitHub — PostgreSQL alternative for evaluation
- VectorDBBench — Open source benchmark to compare vector DBs
- Decision Matrix Template (Notion) — Visual decision matrix template
Estimated time: 30-40 minutes
Next: 05-cost-and-roi-analysis.md