Module 6: Decision Matrix for AI Engineers

Capsule 02: Requirements and Decision Criteria

Learn to turn product needs into measurable criteria that feed your decision matrix. Without clear criteria, any comparison between vector databases is opinion disguised as analysis.

By the end of this capsule:

✅ You'll identify the 6 critical dimensions for evaluating vector databases
✅ You'll translate business requirements into concrete technical thresholds
✅ You'll build a reusable Requirements Document in Python
✅ You'll avoid the most common mistakes when defining criteria

Estimated time: 25-35 minutes

Capsule description

Before opening Pinecone's pricing page or reading Qdrant's README, you need to answer a fundamental question: what does your project need? Most teams jump straight to comparing features without having defined which features matter for their case. The result is a decision based on hype, on the latest viral tweet, or on the inertia of "we use what we already know".

This capsule teaches you a systematic process to extract your project's requirements and turn them into quantifiable decision criteria. It's not about creating a bureaucratic document: it's about having clarity before investing weeks in an integration you'll later have to replace.

The framework you'll build here has 6 dimensions: performance, cost, operational complexity, SDK quality, community/ecosystem, and compliance. Each dimension breaks down into concrete metrics with minimum thresholds. In the end, you'll have a "Requirements Document" in Python that you can reuse in any infrastructure decision.

The 6 evaluation dimensions

Every vector database decision can be broken down into these 6 dimensions. Not all of them weigh the same for every project — that's exactly what you'll resolve in this capsule.

Dimension 1: Performance

Performance is not just "being fast". It breaks down into specific metrics:

performance_criteria = {
    "latency_p50_ms": {
        "description": "Median search latency",
        "why_matters": "Base experience of the average user",
        "typical_ranges": {
            "mvp": "< 500ms",
            "production": "< 200ms",
            "real_time": "< 50ms"
        }
    },
    "latency_p95_ms": {
        "description": "Latency at the 95th percentile",
        "why_matters": "Experience of the worst 5% of requests",
        "typical_ranges": {
            "mvp": "< 1000ms",
            "production": "< 500ms",
            "real_time": "< 100ms"
        }
    },
    "throughput_qps": {
        "description": "Queries per second supported",
        "why_matters": "Concurrent traffic capacity",
        "typical_ranges": {
            "mvp": "10-50 QPS",
            "production": "100-500 QPS",
            "high_traffic": "1000+ QPS"
        }
    },
    "index_build_time": {
        "description": "Index build/update time",
        "why_matters": "Ingestion speed for new documents",
        "typical_ranges": {
            "batch": "< 1 hour for 1M vectors",
            "near_realtime": "< 5 seconds per document",
            "realtime": "< 100ms per document"
        }
    },
    "recall_at_k": {
        "description": "Percentage of correct results in top-K",
        "why_matters": "Quality of the search results",
        "typical_ranges": {
            "acceptable": "> 90%",
            "good": "> 95%",
            "excellent": "> 98%"
        }
    }
}

Key question: Is your application an internal chatbot (latency-tolerant) or a real-time search engine (latency-critical)?

Dimension 2: Cost

Cost has visible and hidden components. Don't make the mistake of comparing only the sticker price:

cost_criteria = {
    "monthly_service_cost": {
        "description": "Monthly cost of the service/infrastructure",
        "components": [
            "Compute (CPU/RAM)",
            "Storage (vectors + metadata)",
            "Network egress",
            "Backups"
        ]
    },
    "cost_per_million_vectors": {
        "description": "Cost normalized per million vectors",
        "why_matters": "Enables a direct comparison between providers",
        "typical_ranges": {
            "free_tier": "$0 (up to 10K-100K vectors)",
            "startup": "$25-100/million/month",
            "enterprise": "$100-500/million/month"
        }
    },
    "cost_predictability": {
        "description": "Is the price predictable or based on variable usage?",
        "why_matters": "Surprises on the monthly bill",
        "models": {
            "flat": "Fixed price per tier (predictable)",
            "usage_based": "Pay per query/storage (variable)",
            "hybrid": "Fixed base + variable by usage"
        }
    },
    "scaling_cost_curve": {
        "description": "How does the cost grow when scaling?",
        "why_matters": "Today's free tier can be $2K/month tomorrow",
        "patterns": {
            "linear": "Cost grows proportionally to volume",
            "step_function": "Jumps when changing tiers",
            "logarithmic": "Volume discount"
        }
    }
}

Key question: How much can you spend TODAY and how much do you expect to spend in 12 months?

Dimension 3: Operational complexity

This dimension is the most underestimated and the one with the most impact on small teams:

ops_complexity_criteria = {
    "setup_time": {
        "description": "Time from zero to the first working query",
        "typical_ranges": {
            "trivial": "< 30 minutes (managed, pip install)",
            "moderate": "1-4 hours (Docker, basic config)",
            "complex": "1-3 days (cluster, networking, security)"
        }
    },
    "maintenance_hours_monthly": {
        "description": "Expected monthly maintenance hours",
        "components": [
            "Monitoring and alerts",
            "Updates and patches",
            "Manual scaling",
            "Backup and recovery",
            "Debugging issues"
        ],
        "typical_ranges": {
            "managed": "1-3 hours/month",
            "self_hosted_simple": "5-10 hours/month",
            "self_hosted_cluster": "15-25 hours/month"
        }
    },
    "sre_required": {
        "description": "Do you need a dedicated SRE/DevOps?",
        "options": {
            "no": "Developers can operate it as a side-task",
            "partial": "DevOps dedicates 20-30% of their time",
            "yes": "Requires a dedicated SRE or infra team"
        }
    },
    "disaster_recovery": {
        "description": "How easy is it to recover from a failure?",
        "factors": [
            "Automatic vs manual backups",
            "Recovery time (RTO)",
            "Acceptable data loss (RPO)",
            "Runbook documentation"
        ]
    }
}

Key question: Does your team have the capacity to operate infrastructure, or is every hour of ops one hour less of product?

Dimension 4: SDK quality and Developer Experience

A bad SDK can cost you more than a high price:

sdk_quality_criteria = {
    "language_support": {
        "description": "Does it support your stack?",
        "critical": ["Python (mandatory for ML/AI)"],
        "important": ["JavaScript/TypeScript", "Go", "Rust"],
        "nice_to_have": ["Java", "C#", "Ruby"]
    },
    "api_design": {
        "description": "Is the API intuitive?",
        "indicators": [
            "CRUD operations in < 5 lines of code",
            "Type hints / working autocompletion",
            "Clear error handling (not generic exceptions)",
            "Native async support"
        ]
    },
    "documentation_quality": {
        "description": "Is the documentation complete and up to date?",
        "checklist": [
            "Quick start that works in < 10 min",
            "Complete API reference",
            "Examples for common use cases",
            "Migration guides between versions",
            "Up-to-date changelog"
        ]
    },
    "testing_support": {
        "description": "Can you test without real infrastructure?",
        "options": {
            "excellent": "In-memory mode for tests",
            "good": "Docker compose for CI/CD",
            "poor": "Requires a real instance to test"
        }
    }
}

Key question: How much time do you spend reading docs and debugging the SDK vs building features?

Dimension 5: Community and ecosystem

The community determines the speed at which you solve problems:

community_criteria = {
    "github_activity": {
        "description": "Repository activity",
        "metrics": [
            "Stars (a proxy for popularity, not quality)",
            "Commits in the last 90 days",
            "Open vs closed issues (ratio)",
            "Average response time to issues"
        ]
    },
    "stackoverflow_presence": {
        "description": "Do you find answers when you search on Google?",
        "why_matters": "If there are no answers on StackOverflow, every bug is a support ticket"
    },
    "integration_ecosystem": {
        "description": "Does it integrate with your existing stack?",
        "key_integrations": [
            "LangChain / LlamaIndex",
            "OpenAI / Anthropic embeddings",
            "Web framework (FastAPI, Django)",
            "Monitoring (Datadog, Prometheus)",
            "CI/CD (GitHub Actions, GitLab CI)"
        ]
    },
    "enterprise_support": {
        "description": "Is paid support available?",
        "options": {
            "community_only": "GitHub issues and Discord only",
            "basic_support": "Email support with an SLA",
            "enterprise": "Dedicated support engineer, guaranteed SLA"
        }
    }
}

Key question: When you have a bug at 2am, will you find the answer in 15 minutes or in 3 days?

Dimension 6: Compliance and security

For many projects this is an eliminatory criterion, not a "nice to have":

compliance_criteria = {
    "data_residency": {
        "description": "Where is the data stored?",
        "options": {
            "any_region": "No geographic restriction",
            "specific_regions": "Data must be in US/EU/LATAM",
            "on_premise": "Data cannot leave the data center"
        }
    },
    "encryption": {
        "description": "What level of encryption do you need?",
        "levels": {
            "basic": "Encryption at rest (AES-256)",
            "transit": "Encryption in transit (TLS 1.2+)",
            "advanced": "Customer-managed encryption keys (CMEK)"
        }
    },
    "audit_logging": {
        "description": "Do you need a record of who accessed which data?",
        "levels": {
            "none": "Not required (MVP, internal project)",
            "basic": "Access logs by API key",
            "full": "Complete audit trail with timestamps and user identity"
        }
    },
    "certifications": {
        "description": "Which certifications does your industry require?",
        "common": ["SOC 2 Type II", "GDPR", "HIPAA", "ISO 27001"],
        "note": "The absence of a required certification is an eliminatory criterion"
    },
    "data_isolation": {
        "description": "How is data isolated between tenants?",
        "models": {
            "shared": "Everyone on the same instance (metadata filters)",
            "namespace": "Logical separation by namespace",
            "dedicated": "Separate instance per tenant"
        }
    }
}

Key question: Do you have regulatory obligations that eliminate providers before you even start comparing?

Translating business requirements into technical ones

The most common mistake is evaluating technology with business language. "It must be fast" is not a criterion. Here's the translation process:

def translate_business_to_technical(business_requirements: list[dict]) -> list[dict]:
    """Convert vague business requirements into measurable technical criteria."""

    translations = {
        "fast_responses": {
            "business": "Premium users need fast responses",
            "technical": [
                {"metric": "latency_p95_ms", "threshold": 250, "unit": "ms"},
                {"metric": "error_rate", "threshold": 1.0, "unit": "%"},
                {"metric": "availability", "threshold": 99.9, "unit": "%"}
            ],
            "rationale": "Premium implies a strict SLA, not just 'fast'"
        },
        "handle_growth": {
            "business": "We expect to grow 10x in the next year",
            "technical": [
                {"metric": "max_vectors", "threshold": 10_000_000, "unit": "vectors"},
                {"metric": "horizontal_scaling", "threshold": True, "unit": "bool"},
                {"metric": "cost_at_10x", "threshold": 5000, "unit": "USD/month"}
            ],
            "rationale": "10x in vectors doesn't mean 10x in cost if you scale well"
        },
        "small_team": {
            "business": "We're 3 developers, we have no DevOps",
            "technical": [
                {"metric": "setup_time_hours", "threshold": 4, "unit": "hours"},
                {"metric": "maintenance_hours_monthly", "threshold": 5, "unit": "hours"},
                {"metric": "sre_required", "threshold": False, "unit": "bool"}
            ],
            "rationale": "No DevOps → managed or trivial self-hosted, never a cluster"
        },
        "regulated_industry": {
            "business": "We work with health data (HIPAA)",
            "technical": [
                {"metric": "hipaa_compliant", "threshold": True, "unit": "bool"},
                {"metric": "encryption_at_rest", "threshold": True, "unit": "bool"},
                {"metric": "audit_logging", "threshold": True, "unit": "bool"},
                {"metric": "data_residency", "threshold": "US", "unit": "region"}
            ],
            "rationale": "HIPAA is eliminatory: if it doesn't comply, don't evaluate anything else"
        },
        "tight_budget": {
            "business": "Maximum $200/month on search infrastructure",
            "technical": [
                {"metric": "monthly_cost_max", "threshold": 200, "unit": "USD"},
                {"metric": "free_tier_vectors", "threshold": 50_000, "unit": "vectors"},
                {"metric": "cost_model", "threshold": "predictable", "unit": "type"}
            ],
            "rationale": "A fixed budget requires predictable pricing, not pay-per-query"
        }
    }

    results = []
    for req in business_requirements:
        key = req.get("type")
        if key in translations:
            translation = translations[key]
            results.append({
                "original": translation["business"],
                "criteria": translation["technical"],
                "rationale": translation["rationale"]
            })
    return results


# Usage example
my_requirements = [
    {"type": "fast_responses"},
    {"type": "small_team"},
    {"type": "tight_budget"}
]

technical_criteria = translate_business_to_technical(my_requirements)
for item in technical_criteria:
    print(f"\nBusiness: {item['original']}")
    print(f"Rationale: {item['rationale']}")
    for c in item["criteria"]:
        print(f"  → {c['metric']}: {c['threshold']} {c['unit']}")

Output:

Business: Premium users need fast responses
Rationale: Premium implies a strict SLA, not just 'fast'
  → latency_p95_ms: 250 ms
  → error_rate: 1.0 %
  → availability: 99.9 %

Business: We're 3 developers, we have no DevOps
Rationale: No DevOps → managed or trivial self-hosted, never a cluster
  → setup_time_hours: 4 hours
  → maintenance_hours_monthly: 5 hours
  → sre_required: False bool

Business: Maximum $200/month on search infrastructure
Rationale: A fixed budget requires predictable pricing, not pay-per-query
  → monthly_cost_max: 200 USD
  → free_tier_vectors: 50000 vectors
  → cost_model: predictable type

Requirements Document Builder

Here's a complete class to build and validate your requirements document:

from dataclasses import dataclass, field
from enum import Enum
from typing import Optional


class Priority(Enum):
    ELIMINATORY = "eliminatory"   # If it doesn't comply, discard immediately
    CRITICAL = "critical"         # High weight in the matrix (4-5)
    IMPORTANT = "important"       # Medium weight (2-3)
    NICE_TO_HAVE = "nice_to_have" # Low weight (1)


@dataclass
class Criterion:
    name: str
    dimension: str
    priority: Priority
    threshold_min: Optional[float | str | bool] = None
    threshold_ideal: Optional[float | str | bool] = None
    unit: str = ""
    notes: str = ""
    weight: int = 0

    def __post_init__(self):
        weight_map = {
            Priority.ELIMINATORY: 5,
            Priority.CRITICAL: 4,
            Priority.IMPORTANT: 2,
            Priority.NICE_TO_HAVE: 1,
        }
        if self.weight == 0:
            self.weight = weight_map[self.priority]


@dataclass
class RequirementsDocument:
    project_name: str
    team_size: int
    budget_monthly_usd: float
    timeline_months: int
    current_vector_count: int
    projected_vector_count_12m: int
    criteria: list[Criterion] = field(default_factory=list)

    def add_criterion(self, criterion: Criterion):
        self.criteria.append(criterion)

    def get_eliminatory(self) -> list[Criterion]:
        return [c for c in self.criteria if c.priority == Priority.ELIMINATORY]

    def get_weighted_criteria(self) -> list[Criterion]:
        return sorted(
            [c for c in self.criteria if c.priority != Priority.ELIMINATORY],
            key=lambda c: c.weight,
            reverse=True
        )

    def validate(self) -> list[str]:
        """Check that the document is complete."""
        issues = []
        if not self.criteria:
            issues.append("No criteria defined")
        if not self.get_eliminatory():
            issues.append("No eliminatory criteria: is really nothing a deal-breaker?")
        if self.budget_monthly_usd <= 0:
            issues.append("Budget not defined")
        if self.projected_vector_count_12m <= self.current_vector_count:
            issues.append("12-month projection <= current: are you sure you're not growing?")

        dimensions_covered = set(c.dimension for c in self.criteria)
        expected = {"performance", "cost", "ops", "sdk", "community", "compliance"}
        missing = expected - dimensions_covered
        if missing:
            issues.append(f"Dimensions without criteria: {missing}")

        return issues

    def summary(self) -> str:
        lines = [
            f"=== Requirements Document: {self.project_name} ===",
            f"Team: {self.team_size} people",
            f"Budget: ${self.budget_monthly_usd}/month",
            f"Vectors: {self.current_vector_count:,} → {self.projected_vector_count_12m:,} (12m)",
            f"Timeline: {self.timeline_months} months",
            "",
            "--- Eliminatory criteria ---",
        ]
        for c in self.get_eliminatory():
            lines.append(f"  ❌ {c.name}: minimum {c.threshold_min} {c.unit}")

        lines.append("")
        lines.append("--- Weighted criteria ---")
        for c in self.get_weighted_criteria():
            lines.append(f"  [{c.weight}] {c.name}: {c.threshold_min} - {c.threshold_ideal} {c.unit}")

        issues = self.validate()
        if issues:
            lines.append("")
            lines.append("--- ⚠️ Issues detected ---")
            for issue in issues:
                lines.append(f"  ⚠️ {issue}")

        return "\n".join(lines)

Example: a startup with a RAG chatbot

doc = RequirementsDocument(
    project_name="RAG Chatbot for an e-commerce SaaS",
    team_size=4,
    budget_monthly_usd=300,
    timeline_months=3,
    current_vector_count=50_000,
    projected_vector_count_12m=500_000
)

doc.add_criterion(Criterion(
    name="p95 latency",
    dimension="performance",
    priority=Priority.CRITICAL,
    threshold_min=500,
    threshold_ideal=200,
    unit="ms"
))

doc.add_criterion(Criterion(
    name="Monthly cost",
    dimension="cost",
    priority=Priority.CRITICAL,
    threshold_min=500,
    threshold_ideal=200,
    unit="USD"
))

doc.add_criterion(Criterion(
    name="No dedicated SRE",
    dimension="ops",
    priority=Priority.ELIMINATORY,
    threshold_min=True,
    unit="bool",
    notes="We have no DevOps, developers must be able to operate it"
))

doc.add_criterion(Criterion(
    name="Python SDK",
    dimension="sdk",
    priority=Priority.ELIMINATORY,
    threshold_min=True,
    unit="bool"
))

doc.add_criterion(Criterion(
    name="LangChain integration",
    dimension="community",
    priority=Priority.IMPORTANT,
    threshold_min=True,
    unit="bool"
))

doc.add_criterion(Criterion(
    name="SOC 2",
    dimension="compliance",
    priority=Priority.NICE_TO_HAVE,
    threshold_min=False,
    threshold_ideal=True,
    unit="bool",
    notes="Not required now, but will be required in 18 months"
))

print(doc.summary())

Output:

=== Requirements Document: RAG Chatbot for an e-commerce SaaS ===
Team: 4 people
Budget: $300/month
Vectors: 50,000 → 500,000 (12m)
Timeline: 3 months

--- Eliminatory criteria ---
  ❌ No dedicated SRE: minimum True bool
  ❌ Python SDK: minimum True bool

--- Weighted criteria ---
  [4] p95 latency: 500 - 200 ms
  [4] Monthly cost: 500 - 200 USD
  [2] LangChain integration: True - None bool
  [1] SOC 2: False - True bool

Prioritization process: adapted MoSCoW

Not all criteria weigh the same. Use this adaptation of the MoSCoW framework to prioritize:

def prioritize_criteria(criteria_list: list[dict]) -> dict:
    """
    Classify criteria using MoSCoW adapted for vector databases.

    Must Have    → Eliminatory (if it doesn't comply, discard)
    Should Have  → Critical (weight 4-5 in the matrix)
    Could Have   → Important (weight 2-3)
    Won't Have   → Exclude from this evaluation
    """
    prioritized = {
        "must_have": [],      # Eliminatory
        "should_have": [],    # Critical for scoring
        "could_have": [],     # Secondary differentiators
        "wont_have": [],      # Out of scope
    }

    decision_rules = {
        "must_have": [
            "If it doesn't meet this criterion, do you discard the option?",
            "Is it a legal/regulatory requirement?",
            "Does your app not work without this?"
        ],
        "should_have": [
            "Does it directly impact the user experience?",
            "Does it impact the monthly operational cost?",
            "Does it relate to the scale of the next 12 months?"
        ],
        "could_have": [
            "Is it a differentiator between two tied options?",
            "Will you need it in 18+ months?",
            "Is it a team preference more than a product requirement?"
        ],
        "wont_have": [
            "Is it a feature you won't use in the next 18 months?",
            "Is it a criterion from another product category?",
            "Would it add noise to the evaluation?"
        ]
    }

    for criterion in criteria_list:
        category = criterion.get("moscow", "could_have")
        prioritized[category].append(criterion["name"])

    return prioritized


# Example
my_criteria = [
    {"name": "Python SDK available", "moscow": "must_have"},
    {"name": "p95 latency < 300ms", "moscow": "must_have"},
    {"name": "Cost < $500/month", "moscow": "should_have"},
    {"name": "Native hybrid search", "moscow": "should_have"},
    {"name": "GraphQL API", "moscow": "wont_have"},
    {"name": "Multi-region replication", "moscow": "could_have"},
    {"name": "SOC 2 Type II", "moscow": "could_have"},
    {"name": "GPU acceleration", "moscow": "wont_have"},
]

result = prioritize_criteria(my_criteria)
for category, items in result.items():
    print(f"\n{category.upper().replace('_', ' ')}:")
    for item in items:
        print(f"  • {item}")

Defining time horizons

A frequent mistake is mixing present requirements with future projections without distinguishing them. Define three horizons:

def define_horizons(
    current_vectors: int,
    growth_rate_monthly: float,
    current_qps: float,
    qps_growth_monthly: float
) -> dict:
    """
    Project requirements across three time horizons.

    Args:
        current_vectors: Current vectors
        growth_rate_monthly: Monthly growth rate (0.1 = 10%)
        current_qps: Current queries per second
        qps_growth_monthly: Monthly QPS growth
    """
    horizons = {}

    for label, months in [("now", 0), ("6_months", 6), ("12_months", 12)]:
        vectors = int(current_vectors * (1 + growth_rate_monthly) ** months)
        qps = current_qps * (1 + qps_growth_monthly) ** months
        horizons[label] = {
            "vectors": vectors,
            "qps": round(qps, 1),
            "storage_gb_estimate": round(vectors * 1536 * 4 / 1e9, 2),
        }

    return horizons


projections = define_horizons(
    current_vectors=100_000,
    growth_rate_monthly=0.15,
    current_qps=20,
    qps_growth_monthly=0.10
)

for horizon, data in projections.items():
    print(f"\n{horizon}:")
    print(f"  Vectors: {data['vectors']:,}")
    print(f"  QPS: {data['qps']}")
    print(f"  Estimated storage: {data['storage_gb_estimate']} GB")

Output:

now:
  Vectors: 100,000
  QPS: 20
  Estimated storage: 0.61 GB

6_months:
  Vectors: 231,306
  QPS: 35.4
  Estimated storage: 1.42 GB

12_months:
  Vectors: 535,025
  QPS: 62.8
  Estimated storage: 3.29 GB

Anti-pattern: criteria that look good but aren't useful

bad_criteria = [
    {
        "criterion": "It must be fast",
        "problem": "There's no number. Is fast 100ms or 2 seconds?",
        "fix": "p95 < 250ms for 1536-dimension queries, top-10"
    },
    {
        "criterion": "It must scale",
        "problem": "Scale to what? 100K vectors or 100M?",
        "fix": "Support 2M vectors in 12 months with latency < 300ms p95"
    },
    {
        "criterion": "Good documentation",
        "problem": "Subjective. What is 'good'?",
        "fix": "Working quick start in < 15 min, API reference with Python examples"
    },
    {
        "criterion": "Cheap",
        "problem": "Compared to what? Does it include operational cost?",
        "fix": "TCO < $400/month including 5h/month of maintenance at $60/hr"
    },
    {
        "criterion": "The one most people use",
        "problem": "Popularity ≠ suitable for your case",
        "fix": "Active community: >50 answers on SO, issues closed in <7 days"
    },
]

for bad in bad_criteria:
    print(f"❌ '{bad['criterion']}'")
    print(f"   Problem: {bad['problem']}")
    print(f"   ✅ Better: '{bad['fix']}'")
    print()

🔧 Troubleshooting

Problem 1: "I have too many criteria and I don't know which ones matter"

Symptom: Your list has 15+ criteria and you can't decide weights.

Solution: Apply the "if it doesn't comply, do you discard?" rule. If the answer is yes, it's eliminatory. If the answer is "it depends", drop it to should/could. Limit your weighted criteria to a maximum of 6 — more than that dilutes the signal.

Problem 2: "The team can't agree on priorities"

Symptom: The CTO wants performance, the PM wants low cost, the developer wants a good SDK.

Solution: Have each stakeholder assign weights independently (without seeing the others'). Average the weights and only discuss differences of more than 2 points. This eliminates the bias of the most vocal person.

Problem 3: "I don't have data to define thresholds"

Symptom: You don't know what latency you need because you never measured.

Solution: Use industry reference benchmarks as a starting point (p95 < 500ms for chatbots, < 100ms for autocomplete). Mark these thresholds as "provisional" and update them after the first PoC with real data.

Problem 4: "The requirements change every week"

Symptom: The product manager brings new requirements constantly and your evaluation never ends.

Solution: Freeze the criteria per evaluation cycle (typically 2-4 weeks). Document new requirements as "v2" but don't incorporate them into the ongoing evaluation. Re-validate quarterly.

Problem 5: "Compliance eliminates almost all my options"

Symptom: HIPAA/SOC2/GDPR reduces your options to 1-2 providers.

Solution: This is correct — compliance is eliminatory by design. If only 1-2 options remain, your evaluation is simpler: compare them against each other or evaluate whether self-hosted with your own compliance is viable.

🏋️ Exercises

Exercise 1: Translate business requirements

Your PM tells you: "We need a search system that's fast, doesn't cost much, that the team can maintain, and that complies with GDPR because we have users in Europe." Translate each phrase into technical criteria with a threshold.

Solution

translated = {
    "fast": {
        "criteria": [
            {"metric": "latency_p95_ms", "threshold": 300},
            {"metric": "latency_p50_ms", "threshold": 150},
        ],
        "rationale": "Without more context, 300ms p95 is a good default for chatbot/search"
    },
    "doesn't cost much": {
        "criteria": [
            {"metric": "monthly_tco_usd", "threshold": 500},
            {"metric": "cost_model", "threshold": "predictable"},
        ],
        "rationale": "Ask the PM for an exact budget. 'Not much' = define a number"
    },
    "the team can maintain it": {
        "criteria": [
            {"metric": "maintenance_hours_monthly", "threshold": 5},
            {"metric": "sre_required", "threshold": False},
            {"metric": "setup_time_hours", "threshold": 4},
        ],
        "rationale": "Without dedicated DevOps, maintenance must be < 5h/month"
    },
    "complies with GDPR": {
        "criteria": [
            {"metric": "data_residency_eu", "threshold": True},
            {"metric": "data_deletion_api", "threshold": True},
            {"metric": "encryption_at_rest", "threshold": True},
            {"metric": "dpa_available", "threshold": True},
        ],
        "rationale": "GDPR is eliminatory: EU region, right to be forgotten, signable DPA"
    }
}

for phrase, data in translated.items():
    print(f"\n'{phrase}':")
    print(f"  Rationale: {data['rationale']}")
    for c in data["criteria"]:
        print(f"  → {c['metric']}: {c['threshold']}")

Exercise 2: Build a Requirements Document

Create a complete RequirementsDocument for this scenario: a fintech startup, a team of 6 developers (1 part-time DevOps), an $800/month budget, 200K current vectors, a projection of 2M in 12 months, SOC 2 regulation required.

Solution

doc = RequirementsDocument(
    project_name="Fintech RAG - Regulatory documents",
    team_size=6,
    budget_monthly_usd=800,
    timeline_months=12,
    current_vector_count=200_000,
    projected_vector_count_12m=2_000_000
)

doc.add_criterion(Criterion(
    name="SOC 2 Type II", dimension="compliance",
    priority=Priority.ELIMINATORY,
    threshold_min=True, unit="bool",
    notes="Required by contract with enterprise customers"
))
doc.add_criterion(Criterion(
    name="Encryption at rest", dimension="compliance",
    priority=Priority.ELIMINATORY,
    threshold_min=True, unit="bool"
))
doc.add_criterion(Criterion(
    name="Support 2M vectors", dimension="performance",
    priority=Priority.ELIMINATORY,
    threshold_min=2_000_000, unit="vectors"
))
doc.add_criterion(Criterion(
    name="p95 latency", dimension="performance",
    priority=Priority.CRITICAL,
    threshold_min=500, threshold_ideal=200, unit="ms"
))
doc.add_criterion(Criterion(
    name="Monthly TCO", dimension="cost",
    priority=Priority.CRITICAL,
    threshold_min=1200, threshold_ideal=600, unit="USD",
    notes="$800 service + part-time DevOps hours"
))
doc.add_criterion(Criterion(
    name="Monthly maintenance", dimension="ops",
    priority=Priority.IMPORTANT,
    threshold_min=15, threshold_ideal=5, unit="hours/month",
    notes="Part-time DevOps: max 15h/month on this project"
))
doc.add_criterion(Criterion(
    name="Async Python SDK", dimension="sdk",
    priority=Priority.IMPORTANT,
    threshold_min=True, unit="bool"
))
doc.add_criterion(Criterion(
    name="LangChain integration", dimension="community",
    priority=Priority.NICE_TO_HAVE,
    threshold_min=True, unit="bool"
))

print(doc.summary())
issues = doc.validate()
print(f"\nValidation: {'✅ No issues' if not issues else '⚠️ ' + str(len(issues)) + ' issues'}")

Exercise 3: Time horizons

Project the requirements for an e-commerce with 80K current vectors, 20% monthly growth, 15 current QPS with 12% monthly growth. In which time horizon do you exceed the typical free tier of 100K vectors?

Solution

import math

current = 80_000
growth = 0.20

months_to_100k = math.log(100_000 / current) / math.log(1 + growth)
print(f"You exceed 100K vectors in {months_to_100k:.1f} months")

months_to_500k = math.log(500_000 / current) / math.log(1 + growth)
print(f"You exceed 500K vectors in {months_to_500k:.1f} months")

months_to_1m = math.log(1_000_000 / current) / math.log(1 + growth)
print(f"You exceed 1M vectors in {months_to_1m:.1f} months")

projections = define_horizons(
    current_vectors=80_000,
    growth_rate_monthly=0.20,
    current_qps=15,
    qps_growth_monthly=0.12
)

for horizon, data in projections.items():
    print(f"\n{horizon}: {data['vectors']:,} vectors, {data['qps']} QPS")

# Result: you exceed 100K in ~1.2 months
# At 20% monthly, you need to plan for a paid tier from the start

Exercise 4: Identify eliminatory criteria

For each scenario, identify which criteria are eliminatory (Must Have) and justify:

Health app with patient data in the EU
A 48-hour hackathon
Multi-tenant SaaS with 50 enterprise customers

Solution

scenarios = {
    "health_app_eu": {
        "eliminatory": [
            "GDPR compliance (patient data in the EU)",
            "HIPAA if it includes US health data",
            "Data residency in the EU",
            "Encryption at rest and in transit",
            "Complete audit logging"
        ],
        "rationale": "A health data breach can cost millions in fines"
    },
    "hackathon_48h": {
        "eliminatory": [
            "Setup in < 15 minutes",
            "Free tier available",
            "Working Python SDK"
        ],
        "rationale": "In a hackathon, time-to-first-query is the only real criterion"
    },
    "saas_multi_tenant": {
        "eliminatory": [
            "Data isolation between tenants",
            "Support 50+ namespaces/collections",
            "API key or auth per tenant",
            "Horizontal scaling without downtime"
        ],
        "rationale": "A data leak between tenants destroys the trust of all your customers"
    }
}

for scenario, data in scenarios.items():
    print(f"\n=== {scenario} ===")
    for criterion in data["eliminatory"]:
        print(f"  ❌ MUST: {criterion}")
    print(f"  Rationale: {data['rationale']}")

Exercise 5: Score an SDK's quality

Evaluate the Python SDK of any vector database you've used (or choose ChromaDB) using this rubric. Assign a score from 0 to 1.0 for each sub-criterion.

Solution (example with ChromaDB)

chromadb_sdk_evaluation = {
    "installation": {
        "score": 1.0,
        "evidence": "pip install chromadb. Works in 30 seconds"
    },
    "first_query_time": {
        "score": 0.9,
        "evidence": "Working hello world in < 5 minutes with the docs"
    },
    "type_hints": {
        "score": 0.7,
        "evidence": "Partial type hints, autocompletion works but isn't perfect"
    },
    "error_messages": {
        "score": 0.6,
        "evidence": "Some errors are cryptic (DimensionalityException without context)"
    },
    "async_support": {
        "score": 0.5,
        "evidence": "AsyncClient exists but isn't prominently documented"
    },
    "testing_support": {
        "score": 0.9,
        "evidence": "EphemeralClient perfect for tests, no external infra"
    },
    "documentation": {
        "score": 0.8,
        "evidence": "Clear docs for basic cases, missing advanced examples"
    }
}

total = sum(v["score"] for v in chromadb_sdk_evaluation.values())
max_score = len(chromadb_sdk_evaluation)
normalized = total / max_score

print(f"ChromaDB SDK Score: {normalized:.2f} / 1.0")
for criterion, data in chromadb_sdk_evaluation.items():
    print(f"  {criterion}: {data['score']} — {data['evidence']}")

🔗 Connection with the project: Decision Questionnaire

In your final project (Decision Questionnaire), the criteria you define here are the input to the whole system. Your questionnaire must:

Ask the user about each dimension (performance, cost, ops, SDK, community, compliance)
Translate the answers into technical criteria with thresholds (using the translate_business_to_technical function)
Generate the RequirementsDocument automatically
Validate that no critical dimensions are missing

The RequirementsDocument and Criterion code from this capsule will be the foundation of your requirements module.

Summary

Define criteria BEFORE evaluating providers. Without clear criteria, any comparison is opinion.
The 6 dimensions cover everything: performance, cost, ops, SDK, community, compliance.
Each criterion needs a concrete number. "Fast" doesn't work; "p95 < 250ms" does.
Eliminatory criteria are evaluated first. If they don't pass, don't waste time scoring.
Distinguish time horizons: what you need today vs 6 vs 12 months.
Team capacity is a technical criterion. Don't ignore the human factor.
Use MoSCoW to prioritize: Must/Should/Could/Won't avoids "everything is important".
Document assumptions. When the requirements change, you'll know what to re-evaluate.

Additional resources

Decision Matrix Analysis (MindTools) — A general framework for decision matrices
MoSCoW Prioritization — A prioritization method by categories
ANN Benchmarks — Performance benchmarks for vector databases
Pinecone Documentation — Example of a managed provider's docs
ChromaDB Documentation — Example of an open-source provider's docs
Qdrant Documentation — Example of a hybrid provider's docs
GDPR Requirements for Data Processors — GDPR requirements for data processors
SOC 2 Compliance Guide — A practical SOC 2 guide

Estimated time: 25-35 minutes Next: 03-scoring-weights-and-the-matrix.md

← PreviousModule 6: Decision Matrix for AI Engineers Next →Capsule 03: Weights, Scoring, and Matrix Methodology

Capsule 02: Requirements and Decision Criteria

🎯 Capsule objective

Capsule description

The 6 evaluation dimensions

Dimension 1: Performance

Dimension 2: Cost

Dimension 3: Operational complexity

Dimension 4: SDK quality and Developer Experience

Dimension 5: Community and ecosystem

Dimension 6: Compliance and security

Translating business requirements into technical ones

Requirements Document Builder

Example: a startup with a RAG chatbot

Prioritization process: adapted MoSCoW

Defining time horizons

Anti-pattern: criteria that look good but aren't useful

🔧 Troubleshooting

Problem 1: "I have too many criteria and I don't know which ones matter"

Problem 2: "The team can't agree on priorities"

Problem 3: "I don't have data to define thresholds"

Problem 4: "The requirements change every week"

Problem 5: "Compliance eliminates almost all my options"

🏋️ Exercises

Exercise 1: Translate business requirements

Exercise 2: Build a Requirements Document

Exercise 3: Time horizons

Exercise 4: Identify eliminatory criteria

Exercise 5: Score an SDK's quality

🔗 Connection with the project: Decision Questionnaire

Summary

Additional resources