Module 6: Decision Matrix for AI Engineers
Capsule 08: Project - Decision Questionnaire
Capsule description
You'll build an interactive questionnaire in Python that gathers a project's requirements, applies the module's weighted scoring matrix, and produces a justified recommendation with an alternative, explicit trade-offs, and a confidence level. It's not a static form: it's an executable tool any AI engineer can run to make defensible decisions.
Estimated time: 35-45 minutes
🎯 Project objective
- Implement an interactive questionnaire with 8 closed-option questions.
- Apply weighted scoring with the formula
score_total = sum(weight * provider_score). - Produce a primary recommendation + alternative with per-criterion justification.
- Validate against 3 different scenarios and generate a readable report.
📋 Project specifications
Functional requirements
- Present questions with numbered options and capture valid answers.
- Map each answer to matrix criteria with configurable weights.
- Calculate a normalized score (0-100) for each provider.
- Produce a top 1 recommendation + top 2 alternative with justification and trade-offs.
Success Criteria
- ✅ Covers 5 providers: ChromaDB, Pinecone, Weaviate, Qdrant, Milvus
- ✅ 8 questions with closed options (no open answers)
- ✅ 3 scenarios validated with a coherent result
- ✅ Readable report without additional context
🧠 Context before starting
This project synthesizes capsules 02-07: decision criteria (weights), matrix scoring (formula), practical matrix (application), cost analysis (budget), recommendation by scenario (validation), and real cases (context).
Difference from Module 5: the decision tree uses branching logic (if/else); this questionnaire uses weighted additive scoring where all criteria contribute proportionally to the final result.
💻 Step-by-step implementation
Step 1: Define the matrix criteria and weights
The criteria come from capsule 02 and the weights from capsule 03. Each weight (1-5) reflects how much that criterion matters in the final decision.
from dataclasses import dataclass, field
CRITERIA_WEIGHTS = {
"scale": {"name": "Expected scale", "weight": 5},
"latency": {"name": "Target latency", "weight": 5},
"ops_simplicity": {"name": "Operational simplicity", "weight": 4},
"cost_control": {"name": "Cost control", "weight": 3},
"compliance": {"name": "Compliance and data control", "weight": 4},
"hybrid_search": {"name": "Hybrid search", "weight": 3},
"multi_tenancy": {"name": "Multi-tenancy", "weight": 2},
"ecosystem": {"name": "Ecosystem and maturity", "weight": 2},
}Why this design? The weights are centralized in a dictionary so you can adjust them without touching the scoring logic.
Step 2: Define provider profiles
Each provider has a score per criterion using the scale from capsule 03: 1.0 (fully meets), 0.7 (meets with a trade-off), 0.4 (partial), 0.1 (not recommended).
@dataclass
class VectorDBProvider:
"""Scoring profile of a vector database provider."""
name: str
scores: dict[str, float]
best_for: list[str] = field(default_factory=list)
risks: list[str] = field(default_factory=list)
typical_cost_range: str = ""
PROVIDERS = {
"ChromaDB": VectorDBProvider(
name="ChromaDB",
scores={"scale": 0.3, "latency": 0.7, "ops_simplicity": 1.0, "cost_control": 1.0,
"compliance": 0.1, "hybrid_search": 0.1, "multi_tenancy": 0.1, "ecosystem": 0.5},
best_for=["PoC and fast prototypes", "local development", "learning"],
risks=["not designed for production at scale", "no official managed offering", "limited features"],
typical_cost_range="$0 (open source)",
),
"Pinecone": VectorDBProvider(
name="Pinecone",
scores={"scale": 0.9, "latency": 0.9, "ops_simplicity": 1.0, "cost_control": 0.4,
"compliance": 0.7, "hybrid_search": 0.8, "multi_tenancy": 0.9, "ecosystem": 0.9},
best_for=["managed production without DevOps", "fast time-to-market", "demanding SLA"],
risks=["strong vendor lock-in", "costs scale with volume", "no self-hosted"],
typical_cost_range="$70-$500+/month",
),
"Weaviate": VectorDBProvider(
name="Weaviate",
scores={"scale": 0.8, "latency": 0.7, "ops_simplicity": 0.6, "cost_control": 0.7,
"compliance": 0.8, "hybrid_search": 1.0, "multi_tenancy": 0.8, "ecosystem": 0.8},
best_for=["advanced hybrid search", "managed + self-hosted flexibility"],
risks=["steeper learning curve", "operational complexity when self-hosted"],
typical_cost_range="$25-$300+/month or your own infra",
),
"Qdrant": VectorDBProvider(
name="Qdrant",
scores={"scale": 0.8, "latency": 0.9, "ops_simplicity": 0.7, "cost_control": 0.8,
"compliance": 0.8, "hybrid_search": 0.8, "multi_tenancy": 0.8, "ecosystem": 0.7},
best_for=["best performance/cost ratio", "self-hosted control with good UX"],
risks=["younger ecosystem", "self-hosted requires operational discipline"],
typical_cost_range="$25-$200+/month or your own infra",
),
"Milvus": VectorDBProvider(
name="Milvus",
scores={"scale": 1.0, "latency": 0.6, "ops_simplicity": 0.2, "cost_control": 0.5,
"compliance": 0.9, "hybrid_search": 0.8, "multi_tenancy": 0.9, "ecosystem": 0.7},
best_for=["massive enterprise scale (>10M)", "team with a dedicated platform"],
risks=["overkill for small projects", "high operational complexity"],
typical_cost_range="$100-$1000+/month or your own infra",
),
}Note: these scores are pedagogical estimates. In a real project, validate against your own benchmarks and up-to-date pricing.
Step 3: Define the questionnaire questions
Each question has closed options and each option adjusts the criteria scores. The mapping between answer and criterion is explicit.
@dataclass
class QuestionOption:
label: str
criteria_adjustments: dict[str, float]
@dataclass
class Question:
id: str
text: str
context: str
options: list[QuestionOption]
QUESTIONS = [
Question(
id="volume",
text="How many vectors do you need to store (12-month projection)?",
context="Include expected growth.",
options=[
QuestionOption("Fewer than 500K vectors", {"scale": 0.3}),
QuestionOption("Between 500K and 5M vectors", {"scale": 0.6}),
QuestionOption("Between 5M and 50M vectors", {"scale": 0.85}),
QuestionOption("More than 50M vectors", {"scale": 1.0}),
],
),
Question(
id="latency",
text="What is your p95 latency requirement?",
context="p95 = 95% of queries must complete within this time.",
options=[
QuestionOption("Flexible (> 100ms is fine)", {"latency": 0.3}),
QuestionOption("Moderate (p95 < 50ms)", {"latency": 0.6}),
QuestionOption("Strict (p95 < 20ms)", {"latency": 0.85}),
QuestionOption("Ultra-low (p95 < 10ms)", {"latency": 1.0}),
],
),
Question(
id="budget",
text="What is your monthly budget for the vector database?",
context="Includes infra + service.",
options=[
QuestionOption("$0 - Free/open source options only", {"cost_control": 1.0}),
QuestionOption("Up to $200/month", {"cost_control": 0.7}),
QuestionOption("$200-$1000/month", {"cost_control": 0.4}),
QuestionOption("More than $1000/month", {"cost_control": 0.2}),
],
),
Question(
id="team",
text="What is your team's operational capacity?",
context="DevOps = ability to manage infra, monitoring, backups.",
options=[
QuestionOption("No DevOps - I need fully managed", {"ops_simplicity": 1.0}),
QuestionOption("Small team with some infra experience", {"ops_simplicity": 0.6}),
QuestionOption("Team with DevOps or a dedicated platform", {"ops_simplicity": 0.3}),
],
),
Question(
id="compliance",
text="What level of compliance do you need?",
context="Compliance = GDPR, HIPAA, SOC2, data residency.",
options=[
QuestionOption("None - non-sensitive data", {"compliance": 0.1}),
QuestionOption("Basic - generic GDPR, good practices", {"compliance": 0.5}),
QuestionOption("Strict - financial/health regulation, residency", {"compliance": 1.0}),
],
),
Question(
id="operation_model",
text="Do you prefer managed (provider's cloud) or self-hosted?",
context="Managed = less operation, more dependency. Self-hosted = more control.",
options=[
QuestionOption("Managed mandatory", {"ops_simplicity": 1.0, "cost_control": 0.4}),
QuestionOption("Self-hosted preferred", {"ops_simplicity": 0.3, "compliance": 0.9}),
QuestionOption("Flexible", {"ops_simplicity": 0.6, "cost_control": 0.6}),
],
),
Question(
id="hybrid_search",
text="Do you need hybrid search (semantic + keyword)?",
context="Combines embeddings with exact-text filters (BM25).",
options=[
QuestionOption("No - semantic search only", {"hybrid_search": 0.1}),
QuestionOption("Would be useful but not critical", {"hybrid_search": 0.5}),
QuestionOption("Yes - it's a product requirement", {"hybrid_search": 1.0}),
],
),
Question(
id="multi_tenancy",
text="Do you need multi-tenant isolation?",
context="Multi-tenant = data from different clients isolated in the same instance.",
options=[
QuestionOption("No - a single project/client", {"multi_tenancy": 0.1}),
QuestionOption("Yes - multiple clients with data isolation", {"multi_tenancy": 1.0}),
],
),
]Why closed options? They eliminate ambiguity. Each answer has a direct mapping to the matrix, which makes the process repeatable and auditable.
Step 4: Weighted scoring engine
Here you apply the formula from capsule 03: score_total = sum(weight * provider_score), normalized to 0-100.
@dataclass
class ScoringResult:
provider_name: str
normalized_score: float
criteria_breakdown: dict[str, dict]
strengths: list[str]
weaknesses: list[str]
@dataclass
class QuestionnaireResult:
answers: dict[str, int]
requirement_profile: dict[str, float]
rankings: list[ScoringResult]
primary: ScoringResult
alternative: ScoringResult
confidence: str
warnings: list[str]
def calculate_requirement_profile(answers: dict[str, int]) -> dict[str, float]:
"""Convert questionnaire answers into a requirement profile (0.0-1.0 per criterion)."""
profile: dict[str, float] = {}
for question in QUESTIONS:
selected = question.options[answers[question.id]]
for criterion, value in selected.criteria_adjustments.items():
profile[criterion] = max(profile.get(criterion, 0), value)
for criterion in CRITERIA_WEIGHTS:
if criterion not in profile:
profile[criterion] = 0.5
return profile
def score_provider(provider: VectorDBProvider, profile: dict[str, float]) -> ScoringResult:
"""Calculate a provider's score against the requirement profile."""
raw_score = 0.0
max_possible = 0.0
breakdown = {}
strengths, weaknesses = [], []
for criterion, config in CRITERIA_WEIGHTS.items():
weight = config["weight"]
provider_score = provider.scores.get(criterion, 0.0)
req_level = profile.get(criterion, 0.5)
effective_weight = weight * (0.5 if req_level < 0.3 else req_level)
contribution = effective_weight * provider_score
raw_score += contribution
max_possible += effective_weight * 1.0
breakdown[criterion] = {
"name": config["name"], "weight": weight,
"provider_score": provider_score, "requirement_level": req_level,
"contribution": round(contribution, 2),
}
if provider_score >= 0.8 and req_level >= 0.6:
strengths.append(f"{config['name']}: score {provider_score}")
elif provider_score <= 0.3 and req_level >= 0.6:
weaknesses.append(f"{config['name']}: score {provider_score} — important weakness")
normalized = (raw_score / max_possible * 100) if max_possible > 0 else 0.0
return ScoringResult(provider.name, round(normalized, 1), breakdown, strengths, weaknesses)
def evaluate_all_providers(answers: dict[str, int]) -> QuestionnaireResult:
"""Evaluate all providers and produce a complete result."""
profile = calculate_requirement_profile(answers)
warnings = _detect_warnings(profile)
results = [score_provider(p, profile) for p in PROVIDERS.values()]
results.sort(key=lambda r: r.normalized_score, reverse=True)
gap = results[0].normalized_score - results[1].normalized_score
confidence = "high" if gap > 15 else ("medium" if gap > 7 else "low")
return QuestionnaireResult(
answers=answers, requirement_profile=profile,
rankings=results, primary=results[0], alternative=results[1],
confidence=confidence, warnings=warnings,
)
def _detect_warnings(profile: dict[str, float]) -> list[str]:
"""Detect contradictory combinations of requirements."""
warnings = []
if profile.get("compliance", 0) >= 0.8 and profile.get("ops_simplicity", 0) >= 0.8:
warnings.append("Strict compliance + managed can be contradictory.")
if profile.get("scale", 0) >= 0.85 and profile.get("cost_control", 0) >= 0.7:
warnings.append("High scale + a tight budget is hard to combine.")
if profile.get("scale", 0) >= 0.85 and profile.get("ops_simplicity", 0) >= 0.8:
warnings.append("High scale with a team without DevOps is an operational risk.")
return warningsDesign decision: effective_weight adjusts the criterion's weight according to how much it matters to the user. If a criterion is not relevant (req_level < 0.3), its weight is halved — this prevents a provider from "winning" just because you ignore its weaknesses.
Step 5: Interactive questionnaire interface
def run_interactive_questionnaire() -> dict[str, int]:
"""Run the interactive questionnaire and return the answers."""
print("=" * 65)
print(" VECTOR DATABASE SELECTION QUESTIONNAIRE")
print(" Module 6 — Decision Matrix for AI Engineers")
print("=" * 65)
print("\nAnswer by selecting the option number.\n")
answers: dict[str, int] = {}
for i, question in enumerate(QUESTIONS, 1):
print(f"─── Question {i}/{len(QUESTIONS)} ───")
print(f"\n {question.text}")
print(f" ({question.context})\n")
for j, option in enumerate(question.options):
print(f" [{j + 1}] {option.label}")
while True:
try:
choice = int(input(f"\n Your answer (1-{len(question.options)}): ").strip()) - 1
if 0 <= choice < len(question.options):
break
print(f" ⚠ Choose a number between 1 and {len(question.options)}")
except ValueError:
print(" ⚠ Enter a valid number")
print(f" ✓ {question.options[choice].label}\n")
answers[question.id] = choice
return answersStep 6: Report generator
The report must be readable without additional context. A stakeholder should understand the recommendation and its justification.
def generate_report(result: QuestionnaireResult) -> str:
"""Generate a complete report of the recommendation."""
lines = []
lines.append("=" * 65)
lines.append(" RECOMMENDATION REPORT — VECTOR DATABASE")
lines.append("=" * 65)
# ── Requirement profile ──
lines.append("\n📋 REQUIREMENT PROFILE")
lines.append("-" * 45)
for criterion, level in sorted(result.requirement_profile.items(), key=lambda x: x[1], reverse=True):
name = CRITERIA_WEIGHTS.get(criterion, {}).get("name", criterion)
bar = "█" * int(level * 10) + "░" * (10 - int(level * 10))
lines.append(f" {name:<30} {bar} {level:.1f}")
if result.warnings:
lines.append(f"\n⚠️ ALERTS ({len(result.warnings)})")
for w in result.warnings:
lines.append(f" ⚠ {w}")
# ── Full ranking ──
lines.append("\n📊 PROVIDER RANKING")
lines.append(f" {'#':<4} {'Provider':<15} {'Score':>8} {'Fit':>10}")
lines.append(" " + "-" * 40)
for rank, r in enumerate(result.rankings, 1):
fit = "excellent" if r.normalized_score >= 80 else ("good" if r.normalized_score >= 65 else ("viable" if r.normalized_score >= 50 else "low"))
marker = " ← recommended" if rank == 1 else ""
lines.append(f" {rank:<4} {r.provider_name:<15} {r.normalized_score:>7.1f}% {fit:>10}{marker}")
# ── Primary recommendation ──
prov = PROVIDERS[result.primary.provider_name]
lines.append(f"\n✅ PRIMARY RECOMMENDATION: {result.primary.provider_name}")
lines.append("-" * 45)
lines.append(f" Score: {result.primary.normalized_score:.1f}% | Confidence: {result.confidence}")
lines.append(f" Typical cost: {prov.typical_cost_range}")
lines.append("\n Strengths for your case:")
for s in (result.primary.strengths[:4] or ["Good overall fit without dominant strengths"]):
lines.append(f" ✓ {s}")
lines.append("\n Risks to consider:")
for risk in prov.risks:
lines.append(f" ✗ {risk}")
# ── Alternative ──
alt = PROVIDERS[result.alternative.provider_name]
lines.append(f"\n🔄 ALTERNATIVE: {result.alternative.provider_name}")
lines.append(f" Score: {result.alternative.normalized_score:.1f}% | Cost: {alt.typical_cost_range}")
lines.append(f"\n Prefer {result.alternative.provider_name} if:")
for c in _get_switch_conditions(result.primary.provider_name, result.alternative.provider_name):
lines.append(f" • {c}")
# ── Top 2 scoring detail ──
lines.append("\n📐 SCORING DETAIL — TOP 2")
lines.append(f" {'Criterion':<25} {'Weight':>5} {'P1':>5} {'P2':>5}")
lines.append(" " + "-" * 42)
for criterion, config in CRITERIA_WEIGHTS.items():
p1 = result.primary.criteria_breakdown[criterion]["provider_score"]
p2 = result.alternative.criteria_breakdown[criterion]["provider_score"]
lines.append(f" {config['name'][:24]:<25} {config['weight']:>4} {p1:>4.1f} {p2:>4.1f}")
lines.append(f" P1={result.primary.provider_name} | P2={result.alternative.provider_name}")
# ── Discarded options ──
lines.append("\n❌ LOWER-FIT OPTIONS")
for r in result.rankings[2:]:
lines.append(f" {r.provider_name} ({r.normalized_score:.1f}%)")
for w in (r.weaknesses[:2] or ["Scored lower than the primary options"]):
lines.append(f" ✗ {w}")
# ── Next steps ──
lines.append("\n📌 NEXT STEPS")
if result.confidence == "low":
lines.append(" 1. Options very close — a comparative PoC is recommended.")
lines.append(" 2. Define success metrics before the test.")
elif result.confidence == "medium":
lines.append(f" 1. Validate {result.primary.provider_name} with a PoC (1-2 weeks).")
lines.append(f" 2. Keep {result.alternative.provider_name} as a documented plan B.")
else:
lines.append(f" 1. Proceed with {result.primary.provider_name} for the current phase.")
lines.append(" 2. Re-evaluate in 3-6 months or when scale changes 3x.")
lines.append("\n" + "=" * 65)
return "\n".join(lines)
def _get_switch_conditions(primary: str, alternative: str) -> list[str]:
"""Conditions for preferring the alternative over the primary."""
switch_map = {
("ChromaDB", "Qdrant"): ["you need to move to production with an SLA", "the volume exceeds 500K vectors"],
("Pinecone", "Qdrant"): ["you want a better performance/cost ratio", "you want a self-hosted option"],
("Pinecone", "Weaviate"): ["hybrid search becomes a central requirement", "you need self-hosted in the future"],
("Qdrant", "Pinecone"): ["you prefer zero-ops without DevOps", "you need a contractually guaranteed SLA"],
("Qdrant", "Weaviate"): ["mature hybrid search is a priority", "you need a broader ecosystem"],
("Weaviate", "Qdrant"): ["ultra-low p95 latency is a priority", "you prefer a simpler API"],
("Milvus", "Weaviate"): ["the scale is under 10M vectors", "you prefer lower operational complexity"],
("Milvus", "Qdrant"): ["the volume doesn't justify the complexity", "your team prefers lightweight operation"],
}
return switch_map.get((primary, alternative), ["scale or compliance requirements change", "the team acquires different operational capacity"])Step 7: Validation scenarios
Define 3 scenarios with predefined answers to verify coherence.
def run_validation_scenarios() -> list[dict]:
"""Run 3 validation scenarios with predefined answers."""
scenarios = [
{"name": "Startup MVP — Fast PoC",
"description": "Small team, no budget, local development",
"answers": {"volume": 0, "latency": 0, "budget": 0, "team": 0,
"compliance": 0, "operation_model": 0, "hybrid_search": 0, "multi_tenancy": 0},
"expected_primary": "ChromaDB"},
{"name": "Growing SaaS — Demanding SLA",
"description": "Validated product, multi-tenant, moderate team",
"answers": {"volume": 1, "latency": 2, "budget": 2, "team": 1,
"compliance": 1, "operation_model": 0, "hybrid_search": 1, "multi_tenancy": 1},
"expected_primary": "Pinecone"},
{"name": "Enterprise — Strict compliance",
"description": "Financial regulation, self-hosted, strong team",
"answers": {"volume": 2, "latency": 2, "budget": 3, "team": 2,
"compliance": 2, "operation_model": 1, "hybrid_search": 2, "multi_tenancy": 1},
"expected_primary": "Qdrant"},
]
results = []
for s in scenarios:
qr = evaluate_all_providers(s["answers"])
results.append({
"name": s["name"], "description": s["description"],
"expected": s["expected_primary"], "actual": qr.primary.provider_name,
"score": qr.primary.normalized_score, "confidence": qr.confidence,
"match": qr.primary.provider_name == s["expected_primary"], "full_result": qr,
})
return resultsStep 8: Main script
Bring it all together into an executable flow with two modes: interactive and validation.
import sys
def main():
print("=" * 65)
print(" VECTOR DB DECISION QUESTIONNAIRE")
print(" Module 6 — Decision Matrix for AI Engineers")
print("=" * 65)
if len(sys.argv) > 1 and sys.argv[1] == "--validate":
run_validation_mode()
else:
run_interactive_mode()
def run_interactive_mode():
print("\n─── MODE: Interactive questionnaire ───\n")
answers = run_interactive_questionnaire()
print("\n⏳ Calculating weighted scoring...\n")
result = evaluate_all_providers(answers)
print(generate_report(result))
print("\nDoes the result make sense? If not, review the weights or provider scores.")
print("Run --validate to check the engine's coherence.\n")
def run_validation_mode():
print("\n─── MODE: Scenario validation ───\n")
results = run_validation_scenarios()
print(f" {'Scenario':<35} {'Expected':<12} {'Actual':<12} {'Score':>6} {'Conf':>8} {'OK':>4}")
print(" " + "-" * 78)
for r in results:
status = "✅" if r["match"] else "⚠️"
print(f" {r['name']:<35} {r['expected']:<12} {r['actual']:<12} "
f"{r['score']:>5.1f}% {r['confidence']:>8} {status:>4}")
passed = sum(1 for r in results if r["match"])
print(f"\n Result: {passed}/{len(results)} scenarios validated")
if passed < len(results):
print("\n ⚠️ Scenarios with a discrepancy:")
for r in results:
if not r["match"]:
print(f" • {r['name']}: expected {r['expected']}, got {r['actual']}")
print("\n─── DETAIL PER SCENARIO ───")
for r in results:
print(f"\n{'─' * 65}")
print(generate_report(r["full_result"]))
print(f"\n{'=' * 65}")
print(f" VALIDATION COMPLETE — {len(PROVIDERS)} providers | {passed}/{len(results)} scenarios")
print(f"{'=' * 65}")
if __name__ == "__main__":
main()📊 Expected output
When you run with --validate, you'll see a report with this structure:
─── MODE: Scenario validation ───
Scenario Expected Actual Score Conf OK
──────────────────────────────────────────────────────────────────────────────────
Startup MVP — Fast PoC ChromaDB ChromaDB 82.3% high ✅
Growing SaaS — Demanding SLA Pinecone Pinecone 78.5% medium ✅
Enterprise — Strict compliance Qdrant Qdrant 76.1% medium ✅
Result: 3/3 scenarios validated
✅ PRIMARY RECOMMENDATION: ChromaDB
─────────────────────────────────────────────
Score: 82.3% | Confidence: high
Typical cost: $0 (open source)
Strengths for your case:
✓ Operational simplicity: score 1.0
✓ Cost control: score 1.0
Risks to consider:
✗ not designed for production at scale
✗ no official managed offering
🔄 ALTERNATIVE: Qdrant
Score: 68.7%
Prefer Qdrant if:
• you need to move to production with an SLA
• the volume exceeds 500K vectors
Deliverables
By the end you should have:
- Executable script with an interactive questionnaire and a validation mode (
--validate). - 3 validated scenarios with a documented coherent result.
- Generated report for at least one scenario with a complete justification.
- Reviewed weights and scores — if you adjusted any value, document why.
🔧 Project troubleshooting
"The questionnaire always recommends the same thing"
Check two things. First, verify that your test scenarios have genuinely different answers — if they all pick similar options, the engine converges. Second, review CRITERIA_WEIGHTS: if a criterion has weight 5 and the provider meets it with 1.0, that single criterion can dominate. Make sure no individual criterion represents more than 25% of the maximum score.
"I don't know how to score some answers"
Use the scale from capsule 03: 1.0 = fully meets, 0.7 = meets with a trade-off, 0.4 = partial, 0.1 = not recommended. If you don't have data, assign 0.5 (neutral) and document that it's a pending assumption.
"Stakeholders don't trust the recommendation"
Share: (1) the weights and where they come from, (2) the per-provider scores with their source, (3) the validation scenarios with results. If a stakeholder questions a weight, adjust it together and re-run — the engine is deterministic.
"Two providers tie on score"
This is information, not an error. "Low" confidence indicates you need more data. Three options: add a differentiating criterion, run a comparative PoC with both finalists, or apply the rule from capsule 06: on a tie, choose the operationally simpler one.
"I want to change the weights for my context"
That's what the centralized CRITERIA_WEIGHTS is for. Change the weights, re-run --validate, and check that the scenarios remain coherent. If a weight change breaks a scenario, you have a trade-off to document.
🏋️ Post-project exercises
Exercise 1: Add a vendor lock-in criterion
Extend the questionnaire with a question about portability concern and a new criterion in the matrix.
Hints:
- Add a
"lock_in"entry toCRITERIA_WEIGHTS. - Add lock-in scores to each
VectorDBProvider. - Create a new
Questionwith options that map to"lock_in".
See solution
# 1. Add the criterion:
CRITERIA_WEIGHTS["lock_in"] = {"name": "Vendor lock-in risk", "weight": 3}
# 2. Add scores to each provider (in its "scores" dict):
# ChromaDB: "lock_in": 1.0 (open source, no lock-in)
# Pinecone: "lock_in": 0.2 (proprietary API, no self-hosted)
# Weaviate: "lock_in": 0.8 (open source, standard API)
# Qdrant: "lock_in": 0.8 (open source, standard gRPC)
# Milvus: "lock_in": 0.8 (open source, standard API)
# 3. Add the question to QUESTIONS:
Question(
id="lock_in",
text="How important is it to avoid vendor lock-in?",
context="Lock-in = difficulty of migrating to another provider in the future.",
options=[
QuestionOption("I'm not worried - I prioritize speed", {"lock_in": 0.1}),
QuestionOption("Moderate - I want options but not urgently", {"lock_in": 0.5}),
QuestionOption("High - I need to migrate without rewriting", {"lock_in": 1.0}),
],
)
# 4. Re-run --validate to verify coherenceExercise 2: Export the result to Markdown
Create a function that converts the QuestionnaireResult into a Markdown document ready to share in a PR or decision document.
Hints:
- The function takes a
QuestionnaireResultand returns a Markdown string. - Use Markdown tables for the ranking and the scoring detail.
- Include an assumptions section and an expiration date.
See solution
from datetime import date
def export_to_markdown(result: QuestionnaireResult, project_name: str = "Project") -> str:
"""Export the questionnaire result to Markdown."""
lines = []
lines.append(f"# Vector Database Decision — {project_name}")
lines.append(f"\n**Date:** {date.today().isoformat()}")
lines.append(f"**Confidence:** {result.confidence}")
lines.append(f"**Suggested validity:** 3 months or until a 2x scale change")
lines.append("\n## Ranking\n")
lines.append("| # | Provider | Score | Fit |")
lines.append("|---|----------|------:|-----|")
for rank, r in enumerate(result.rankings, 1):
fit = "excellent" if r.normalized_score >= 80 else ("good" if r.normalized_score >= 65 else "viable")
lines.append(f"| {rank} | {r.provider_name} | {r.normalized_score:.1f}% | {fit} |")
prov = PROVIDERS[result.primary.provider_name]
lines.append(f"\n## Recommendation: {result.primary.provider_name}\n")
lines.append(f"**Typical cost:** {prov.typical_cost_range}\n")
lines.append("### Strengths\n")
for s in result.primary.strengths:
lines.append(f"- {s}")
lines.append("\n### Risks\n")
for risk in prov.risks:
lines.append(f"- {risk}")
lines.append(f"\n## Alternative: {result.alternative.provider_name}\n")
lines.append(f"Score: {result.alternative.normalized_score:.1f}%\n")
lines.append("Consider if:")
for c in _get_switch_conditions(result.primary.provider_name, result.alternative.provider_name):
lines.append(f"- {c}")
lines.append("\n## Scoring detail\n")
lines.append("| Criterion | Weight | P1 | P2 |")
lines.append("|-----------|-------:|---:|---:|")
for criterion, config in CRITERIA_WEIGHTS.items():
p1 = result.primary.criteria_breakdown[criterion]["provider_score"]
p2 = result.alternative.criteria_breakdown[criterion]["provider_score"]
lines.append(f"| {config['name']} | {config['weight']} | {p1:.1f} | {p2:.1f} |")
if result.warnings:
lines.append("\n## Alerts\n")
for w in result.warnings:
lines.append(f"- ⚠️ {w}")
lines.append("\n---\n*Generated with the Vector DB Decision Questionnaire — Module 6*")
return "\n".join(lines)
# Usage:
# md = export_to_markdown(result, "My RAG Project")
# with open("decision-vectordb.md", "w") as f:
# f.write(md)Exercise 3: Weight sensitivity analysis
Create a function that varies each weight in CRITERIA_WEIGHTS (±1) and shows which ones change the recommendation — identifying the most sensitive criteria.
Hints:
- Clone
CRITERIA_WEIGHTS, modify one weight at a time, recalculate scores. - Compare whether the
primarychanges with each variation. - The criteria that change the recommendation are the most sensitive.
See solution
import copy
def sensitivity_analysis(answers: dict[str, int]) -> str:
"""Identify which weights are most sensitive to the result."""
global CRITERIA_WEIGHTS
base_result = evaluate_all_providers(answers)
base_primary = base_result.primary.provider_name
original_weights = copy.deepcopy(CRITERIA_WEIGHTS)
lines = []
lines.append("=" * 65)
lines.append(f" SENSITIVITY ANALYSIS — Base: {base_primary} ({base_result.primary.normalized_score:.1f}%)")
lines.append("=" * 65)
sensitive = []
for criterion, config in original_weights.items():
original_w = config["weight"]
lines.append(f"\n📊 {config['name']} (base weight: {original_w})")
lines.append(f" {'Weight':<8} {'Recommendation':<15} {'Score':>7} {'Changed?':>8}")
lines.append(" " + "-" * 42)
changed = False
for delta in [-2, -1, 0, 1, 2]:
new_w = max(1, min(5, original_w + delta))
CRITERIA_WEIGHTS[criterion]["weight"] = new_w
varied = evaluate_all_providers(answers)
is_diff = varied.primary.provider_name != base_primary
if is_diff:
changed = True
marker = "⚠️ YES" if is_diff else " no"
current = " ←" if delta == 0 else ""
lines.append(f" {new_w:<8} {varied.primary.provider_name:<15} "
f"{varied.primary.normalized_score:>6.1f}% {marker:>8}{current}")
CRITERIA_WEIGHTS[criterion]["weight"] = original_w
if changed:
sensitive.append(config["name"])
CRITERIA_WEIGHTS = original_weights
if sensitive:
lines.append(f"\n⚠️ Sensitive criteria:")
for c in sensitive:
lines.append(f" • {c}")
lines.append("You should estimate these weights more carefully.")
else:
lines.append("\n✅ Robust decision — no individual weight changes the recommendation.")
return "\n".join(lines)
# Usage:
# print(sensitivity_analysis(answers))The output will show which fields marked with "YES" are the most sensitive inputs — the ones you should estimate more carefully.
✅ Completeness checklist
Code structure:
-
CRITERIA_WEIGHTShas 8 criteria with weights from 1 to 5. -
PROVIDERScovers 5 providers with per-criterion scores. -
QUESTIONShas 8 questions with closed options. - Each option maps to matrix criteria.
Scoring engine:
-
score_provider()appliessum(weight * score)normalized to 0-100. -
evaluate_all_providers()ranks providers and calculates confidence. -
_detect_warnings()identifies contradictory combinations.
Decision quality:
- The justification explains strengths (why yes) and risks (watch out for).
- The alternative includes conditions for when to prefer it.
- Per-criterion scoring detail is visible in the report.
Validation:
- 3 scenarios run (MVP, SaaS, Enterprise).
- Discrepancies between expected and actual explained.
-
--validateproduces readable and coherent output.
Output:
- Report readable without additional context.
- Full ranking of 5 providers visible.
- Next steps adjusted to the confidence level.
Summary
- You built an interactive questionnaire that transforms answers into weighted scoring over 5 vector database providers.
- Each question has closed options that map directly to the module's matrix criteria.
- The engine applies the formula
score_total = sum(weight * score)with adjustable weights and normalization to 0-100. - The report produces a recommendation + alternative with strengths, risks, scoring detail, and switch conditions.
- You validated with 3 scenarios (MVP, SaaS, Enterprise) that cover representative requirement combinations.
- The confidence level (high/medium/low) indicates when the decision is clear vs. when you need more analysis.
- The centralized weights let you adapt the tool without modifying the engine's logic.
- This project closes Module 6 and connects with Module 7 (production considerations).
📚 Additional resources
- Decision Matrix — Wikipedia
- Weighted Sum Model — Wikipedia
- Pinecone — Documentation and pricing
- Weaviate — Documentation for developers
- Qdrant — Technical documentation
- Milvus — Documentation and Zilliz Cloud
- ChromaDB — Documentation
- Martin Fowler — Trade-offs in technical decisions
Estimated time: 35-45 minutes
Next module: ../../module-07-production-considerations/en/01-module-introduction-7.md