Module 7: Technical comparison of providers

Project: Decision Tool

Next →

You've reached the end of the technical comparison module. You're going to build an automated Python tool that materializes everything you learned: it takes a requirements profile as input (volume, target latency, budget, constraints, the importance of each dimension) and emits a recommendation with quantitative justification.

It's the deliverable you take to your portfolio and to Module 8. Anyone on the team (or you yourself in 3 months) can run it, pass it a different profile, and get the answer without redoing the analysis from scratch.

By the end of this project you'll have:

  • A Python CLI that ranks providers given a usage profile
  • Declarative configuration (YAML/JSON) to register new providers without touching code
  • A detailed report with score, per-dimension breakdown, and qualitative reasons
  • Tests that validate the filtering, normalization and scoring logic
  • Documentation on how to extend it for your specific product

Specification

Inputs

A product profile file, example product_profile.yaml:

name: "B2B SaaS support chatbot"
monthly_volume: 200000
avg_input_tokens: 600
avg_output_tokens: 250
warm_pool_hours_month: 200

constraints:
  data_residency_eu: false
  soc2_required: true
  budget_max_usd: 1000
  hipaa_required: false
  latency_p95_max_s: 5.0

weights:
  latency: 0.30
  cost: 0.20
  quality: 0.50

A provider catalog, providers.yaml:

providers:
  - name: "OpenAI GPT-4o-mini"
    type: "managed"
    latency_p95_s: 2.3
    price_input_per_1m: 0.15
    price_output_per_1m: 0.60
    quality_score: 86
    meets_soc2: true
    meets_hipaa_baa: true
    meets_data_residency_eu: false
    notes: "Safe default, medium lock-in"

  - name: "Anthropic Claude 3.5 Sonnet"
    type: "managed"
    latency_p95_s: 2.8
    price_input_per_1m: 3.00
    price_output_per_1m: 15.00
    quality_score: 92
    meets_soc2: true
    meets_hipaa_baa: true
    meets_data_residency_eu: false
    notes: "Superior reasoning; expensive"

  - name: "OpenRouter Mistral 7B"
    type: "managed"
    latency_p95_s: 3.0
    price_input_per_1m: 0.07
    price_output_per_1m: 0.07
    quality_score: 69
    meets_soc2: false
    meets_hipaa_baa: false
    meets_data_residency_eu: false
    notes: "Low cost; limited quality"

  - name: "Modal Mistral 7B (A10G + warm)"
    type: "serverless_gpu"
    latency_p95_s: 2.7
    price_gpu_per_second: 0.000306
    gpu_seconds_per_request: 2.0
    quality_score: 69
    meets_soc2: true
    meets_hipaa_baa: false
    meets_data_residency_eu: true
    notes: "Model control; cold starts to mitigate"

  - name: "Self-hosted Ollama Mistral (EU)"
    type: "selfhosted"
    latency_p95_s: 5.5
    price_vm_per_hour: 1.10
    quality_score: 69
    meets_soc2: true       # depends on your compliance
    meets_hipaa_baa: true
    meets_data_residency_eu: true
    notes: "Strict compliance; requires DevOps"

Output

A well-formatted CLI report with:

  1. Summary of the input profile
  2. List of providers discarded by constraints (with reason)
  3. Ranking with normalized scores
  4. Top 1 with the score breakdown
  5. Additional qualitative reasons
  6. Basic sensitivity analysis

Implementation

Create decision_tool.py:

# decision_tool.py
"""
Decision Tool for LLM provider selection.

Usage:
    python decision_tool.py --profile product_profile.yaml --providers providers.yaml
"""
import argparse
import json
import sys
from dataclasses import dataclass, field
from pathlib import Path
import yaml


# ============================================================
# Data models
# ============================================================
@dataclass
class Provider:
    name: str
    type: str
    latency_p95_s: float
    quality_score: float
    meets_soc2: bool = True
    meets_hipaa_baa: bool = True
    meets_data_residency_eu: bool = False
    notes: str = ""
    # Pricing fields - used depending on type
    price_input_per_1m: float | None = None
    price_output_per_1m: float | None = None
    price_gpu_per_second: float | None = None
    gpu_seconds_per_request: float | None = None
    price_vm_per_hour: float | None = None

    @classmethod
    def from_dict(cls, d: dict) -> "Provider":
        return cls(**{k: v for k, v in d.items() if k in cls.__annotations__})

    def estimated_monthly_cost(
        self,
        requests_month: int,
        input_tokens: int,
        output_tokens: int,
        warm_pool_hours: int,
    ) -> float:
        if self.type == "managed":
            return (
                requests_month * input_tokens * self.price_input_per_1m / 1_000_000
                + requests_month * output_tokens * self.price_output_per_1m / 1_000_000
            )
        elif self.type == "serverless_gpu":
            gpu_active = requests_month * self.gpu_seconds_per_request * self.price_gpu_per_second
            warm = warm_pool_hours * 3600 * self.price_gpu_per_second
            storage = 2.0
            return gpu_active + warm + storage
        elif self.type == "selfhosted":
            return 720 * self.price_vm_per_hour
        else:
            raise ValueError(f"Unknown type: {self.type}")


@dataclass
class Profile:
    name: str
    monthly_volume: int
    avg_input_tokens: int
    avg_output_tokens: int
    warm_pool_hours_month: int
    constraints: dict
    weights: dict


# ============================================================
# Loading
# ============================================================
def load_profile(path: Path) -> Profile:
    with open(path) as f:
        data = yaml.safe_load(f)
    return Profile(
        name=data["name"],
        monthly_volume=data["monthly_volume"],
        avg_input_tokens=data["avg_input_tokens"],
        avg_output_tokens=data["avg_output_tokens"],
        warm_pool_hours_month=data.get("warm_pool_hours_month", 0),
        constraints=data.get("constraints", {}),
        weights=data["weights"],
    )


def load_providers(path: Path) -> list[Provider]:
    with open(path) as f:
        data = yaml.safe_load(f)
    return [Provider.from_dict(p) for p in data["providers"]]


# ============================================================
# Filtering by constraints
# ============================================================
def filter_providers(providers: list[Provider], profile: Profile) -> tuple[list[Provider], list[tuple[str, list[str]]]]:
    c = profile.constraints
    valid = []
    discarded = []
    for p in providers:
        reasons = []
        if c.get("data_residency_eu") and not p.meets_data_residency_eu:
            reasons.append("fails EU data residency")
        if c.get("soc2_required") and not p.meets_soc2:
            reasons.append("fails SOC2")
        if c.get("hipaa_required") and not p.meets_hipaa_baa:
            reasons.append("fails HIPAA BAA")
        if c.get("latency_p95_max_s") and p.latency_p95_s > c["latency_p95_max_s"]:
            reasons.append(f"P95 latency {p.latency_p95_s}s > max {c['latency_p95_max_s']}s")

        cost = p.estimated_monthly_cost(
            profile.monthly_volume,
            profile.avg_input_tokens,
            profile.avg_output_tokens,
            profile.warm_pool_hours_month,
        )
        if c.get("budget_max_usd") and cost > c["budget_max_usd"]:
            reasons.append(f"cost ${cost:.0f} > budget ${c['budget_max_usd']:.0f}")

        if reasons:
            discarded.append((p.name, reasons))
        else:
            valid.append(p)
    return valid, discarded


# ============================================================
# Normalization and scoring
# ============================================================
def normalize(values: list[float], lower_is_better: bool) -> list[float]:
    if not values or max(values) == min(values):
        return [1.0] * len(values)
    vmin, vmax = min(values), max(values)
    if lower_is_better:
        return [1 - (v - vmin) / (vmax - vmin) for v in values]
    return [(v - vmin) / (vmax - vmin) for v in values]


def score_providers(providers: list[Provider], profile: Profile) -> list[dict]:
    if not providers:
        return []

    latencies = [p.latency_p95_s for p in providers]
    costs = [
        p.estimated_monthly_cost(
            profile.monthly_volume,
            profile.avg_input_tokens,
            profile.avg_output_tokens,
            profile.warm_pool_hours_month,
        )
        for p in providers
    ]
    qualities = [p.quality_score for p in providers]

    n_lat = normalize(latencies, lower_is_better=True)
    n_cost = normalize(costs, lower_is_better=True)
    n_qual = normalize(qualities, lower_is_better=False)

    weights = profile.weights
    results = []
    for p, lat, cost, qual, nl, nc, nq in zip(
        providers, latencies, costs, qualities, n_lat, n_cost, n_qual
    ):
        score = nl * weights["latency"] + nc * weights["cost"] + nq * weights["quality"]
        results.append(
            {
                "provider": p,
                "latency_s": lat,
                "cost_monthly_usd": cost,
                "quality_score": qual,
                "n_latency": nl,
                "n_cost": nc,
                "n_quality": nq,
                "contribution_latency": nl * weights["latency"],
                "contribution_cost": nc * weights["cost"],
                "contribution_quality": nq * weights["quality"],
                "score_total": score,
            }
        )
    return sorted(results, key=lambda x: x["score_total"], reverse=True)


# ============================================================
# Sensitivity analysis
# ============================================================
def sensitivity(providers: list[Provider], profile: Profile) -> dict:
    """Does the winner change if we move weights ±0.1?"""
    base_weights = profile.weights.copy()
    base = score_providers(providers, profile)
    if not base:
        return {"stable": False, "variations": []}

    base_winner = base[0]["provider"].name
    variations = []
    for dim, delta in [("latency", 0.1), ("latency", -0.1), ("cost", 0.1), ("cost", -0.1), ("quality", 0.1), ("quality", -0.1)]:
        new_weights = base_weights.copy()
        new_weights[dim] = max(0, min(1, new_weights[dim] + delta))
        # Renormalize so they sum to 1
        total = sum(new_weights.values())
        new_weights = {k: v / total for k, v in new_weights.items()}
        temp_profile = Profile(
            **{**profile.__dict__, "weights": new_weights}
        )
        r = score_providers(providers, temp_profile)
        if r:
            variations.append(
                {"change": f"{dim} {'+' if delta > 0 else ''}{delta}", "winner": r[0]["provider"].name}
            )

    distinct = {v["winner"] for v in variations}
    return {"stable": len(distinct) == 1 and base_winner in distinct, "variations": variations}


# ============================================================
# Report
# ============================================================
def report(profile: Profile, discarded, ranking, sens):
    sep = "=" * 80
    print(f"\n{sep}")
    print(f"  DECISION TOOL — {profile.name}")
    print(f"{sep}\n")

    print(f"📋 Usage profile:")
    print(f"   Volume: {profile.monthly_volume:,} req/month")
    print(f"   Tokens: {profile.avg_input_tokens} input + {profile.avg_output_tokens} output")
    print(f"   Warm pool: {profile.warm_pool_hours_month} hrs/month")
    print(f"\n⚖️  Weights: lat={profile.weights['latency']}, cost={profile.weights['cost']}, qual={profile.weights['quality']}")

    if discarded:
        print(f"\n❌ Discarded ({len(discarded)}):")
        for name, reasons in discarded:
            print(f"   {name}: {'; '.join(reasons)}")

    if not ranking:
        print("\n⚠️  No provider meets the constraints. Review the profile.")
        return

    print(f"\n📊 Ranking ({len(ranking)} candidates):")
    print(f"   {'Provider':<40} {'P95':>6} {'$/mo':>10} {'Qual':>5} {'Score':>7}")
    print(f"   {'-' * 70}")
    for r in ranking:
        p = r["provider"]
        print(f"   {p.name:<40} {r['latency_s']:>5.2f}s {r['cost_monthly_usd']:>9.0f} {r['quality_score']:>4.0f} {r['score_total']:>7.3f}")

    top = ranking[0]
    p = top["provider"]
    print(f"\n🏆 Recommendation: {p.name}")
    print(f"   Score total: {top['score_total']:.3f}")
    print(f"   Breakdown:")
    print(f"     Latency:  {top['contribution_latency']:.3f} (norm {top['n_latency']:.2f})")
    print(f"     Cost:     {top['contribution_cost']:.3f} (norm {top['n_cost']:.2f})")
    print(f"     Quality:  {top['contribution_quality']:.3f} (norm {top['n_quality']:.2f})")
    if p.notes:
        print(f"   Notes: {p.notes}")

    print(f"\n🔬 Sensitivity analysis (±0.1 movement in weights):")
    if sens["stable"]:
        print(f"   ✅ Robust decision — the winner doesn't change with weight variations")
    else:
        print(f"   ⚠️  Decision sensitive to weights — the winner changes with weighting:")
        for v in sens["variations"]:
            print(f"     {v['change']:<15} → {v['winner']}")


# ============================================================
# CLI
# ============================================================
def main():
    parser = argparse.ArgumentParser(description="Decision tool for LLM provider selection")
    parser.add_argument("--profile", required=True, type=Path, help="YAML with the product profile")
    parser.add_argument("--providers", required=True, type=Path, help="YAML with the provider catalog")
    parser.add_argument("--json", action="store_true", help="Output as JSON")
    args = parser.parse_args()

    profile = load_profile(args.profile)
    providers = load_providers(args.providers)

    valid, discarded = filter_providers(providers, profile)
    ranking = score_providers(valid, profile)
    sens = sensitivity(valid, profile)

    if args.json:
        print(json.dumps(
            {
                "profile": profile.name,
                "ranking": [
                    {
                        "provider": r["provider"].name,
                        "score": r["score_total"],
                        "cost_monthly": r["cost_monthly_usd"],
                        "latency_p95": r["latency_s"],
                        "quality": r["quality_score"],
                    }
                    for r in ranking
                ],
                "discarded": [{"name": n, "reasons": r} for n, r in discarded],
                "sensitivity_stable": sens["stable"],
            },
            indent=2,
        ))
    else:
        report(profile, discarded, ranking, sens)


if __name__ == "__main__":
    main()

Tests

Create test_decision_tool.py:

# test_decision_tool.py
import pytest
from decision_tool import Provider, Profile, filter_providers, score_providers, normalize, sensitivity


def basic_profile():
    return Profile(
        name="test",
        monthly_volume=100_000,
        avg_input_tokens=500,
        avg_output_tokens=200,
        warm_pool_hours_month=0,
        constraints={"budget_max_usd": 500},
        weights={"latency": 0.3, "cost": 0.4, "quality": 0.3},
    )


def test_normalize_lower_is_better():
    n = normalize([1.0, 2.0, 4.0], lower_is_better=True)
    assert n[0] == 1.0   # lower = better = 1
    assert n[2] == 0.0   # higher = worse = 0
    assert 0 < n[1] < 1  # intermediate


def test_normalize_equal_returns_ones():
    n = normalize([5.0, 5.0], lower_is_better=True)
    assert n == [1.0, 1.0]


def test_filter_by_budget():
    p_expensive = Provider(
        name="Expensive", type="managed", latency_p95_s=2.0, quality_score=90,
        price_input_per_1m=10, price_output_per_1m=30,
    )
    p_cheap = Provider(
        name="Cheap", type="managed", latency_p95_s=3.0, quality_score=70,
        price_input_per_1m=0.1, price_output_per_1m=0.5,
    )
    valid, discarded = filter_providers([p_expensive, p_cheap], basic_profile())
    assert len(valid) == 1
    assert valid[0].name == "Cheap"
    assert any("budget" in r for _, reasons in discarded for r in reasons)


def test_filter_by_data_residency():
    profile = basic_profile()
    profile.constraints = {"data_residency_eu": True}
    p_us = Provider(name="US", type="managed", latency_p95_s=2, quality_score=80,
                    price_input_per_1m=0.1, price_output_per_1m=0.1,
                    meets_data_residency_eu=False)
    p_eu = Provider(name="EU", type="managed", latency_p95_s=2, quality_score=80,
                    price_input_per_1m=0.1, price_output_per_1m=0.1,
                    meets_data_residency_eu=True)
    valid, _ = filter_providers([p_us, p_eu], profile)
    assert {v.name for v in valid} == {"EU"}


def test_scoring_rewards_best_at_everything():
    """A provider strictly better at everything should win."""
    p1 = Provider(name="Best", type="managed", latency_p95_s=1.0, quality_score=95,
                  price_input_per_1m=0.05, price_output_per_1m=0.05)
    p2 = Provider(name="Worst", type="managed", latency_p95_s=5.0, quality_score=60,
                  price_input_per_1m=5.0, price_output_per_1m=15.0)
    ranking = score_providers([p1, p2], basic_profile())
    assert ranking[0]["provider"].name == "Best"


def test_sensitivity_detects_instability():
    # Two providers close in score → HIGH sensitivity
    p1 = Provider(name="A", type="managed", latency_p95_s=2.0, quality_score=85,
                  price_input_per_1m=0.3, price_output_per_1m=0.6)
    p2 = Provider(name="B", type="managed", latency_p95_s=2.1, quality_score=84,
                  price_input_per_1m=0.3, price_output_per_1m=0.6)
    # Near-twins — some weight changes should change the winner or not
    sens = sensitivity([p1, p2], basic_profile())
    # No strict assert — we just verify the function returns a valid structure
    assert "stable" in sens
    assert "variations" in sens

Run:

pip install pytest pyyaml
pytest test_decision_tool.py -v

Real usage

# Case 1: B2B SaaS standard
python decision_tool.py --profile product_profile.yaml --providers providers.yaml

# Case 2: client with HIPAA
cat > hipaa_profile.yaml <<EOF
name: "Medical assistant"
monthly_volume: 50000
avg_input_tokens: 800
avg_output_tokens: 300
warm_pool_hours_month: 100
constraints:
  hipaa_required: true
  soc2_required: true
  budget_max_usd: 2000
weights:
  latency: 0.2
  cost: 0.2
  quality: 0.6
EOF

python decision_tool.py --profile hipaa_profile.yaml --providers providers.yaml

# Case 3: JSON output to integrate into a pipeline
python decision_tool.py --profile product_profile.yaml --providers providers.yaml --json > recommendation.json

Suggested README

# LLM Provider Decision Tool

Recommends the optimal LLM provider given your usage profile.

## Installation

```bash
pip install pyyaml
```

## Usage

```bash
python decision_tool.py --profile your_profile.yaml --providers providers.yaml
```

## File structure

- `product_profile.yaml`: your product's requirements and priorities
- `providers.yaml`: provider catalog with their metrics
- `decision_tool.py`: the script

## Extend

To add a provider: edit `providers.yaml`. Supported types:
- `managed`: pricing per tokens (OpenAI, Anthropic, OpenRouter)
- `serverless_gpu`: pricing per GPU second (Modal, Replicate)
- `selfhosted`: pricing per VM hour (AWS, GCP)

To change weights: edit `product_profile.yaml`. The weights must sum to 1.0.

## Tests

```bash
pytest test_decision_tool.py -v
```

## Important

The provider numbers in `providers.yaml` are **references**. Replace them with your real benchmarks (capsules 02-04 of the path) before using in production.

Self-assessment

Before closing the module:

  • The script runs without errors with the example files
  • Changing weights in product_profile.yaml changes the ranking
  • Enabling constraints (hipaa_required, data_residency_eu) discards providers correctly
  • The tests pass (pytest -v)
  • The JSON output with --json is valid and parseable
  • The sensitivity analysis detects when there's a close tie

Connection with the rest of the path

What you built is the qualitative input of Module 8 (Unified Client):

  • Your decision tool says "OpenAI wins in this profile"
  • The M08 Unified Client lets you switch providers with one parameter
  • When your profile changes (prices, constraints), you run decision tool → get a new winner → switch the provider in Unified Client → near-zero migration

Your work here doesn't end with M07. It starts taking real value in M08.

Module 7 completed

You did something uncommon: you turned LLM provider selection into a reproducible, defensible process. You'll be able to show this decision tool in interviews or use your own output in real product decisions.

Concretely, you can now:

  • ✅ Benchmark latency with real P50/P95/P99
  • ✅ Calculate honest costs with hidden costs included
  • ✅ Measure quality with a rubric + LLM-as-judge
  • ✅ Combine dimensions in a weighted matrix with sensitivity
  • ✅ Migrate between providers knowing the cost of each migration
  • ✅ Rank options with an automated CLI tool

Next module

Module 8 — Unified AI Client is the close of the path. You're going to build the client that encapsulates all the providers you got to know behind a single interface, with automatic fallback, declarative configuration, and multi-provider testing. It's the deliverable you take to your portfolio as proof of mastery of the topic.

Resources

  1. TOPSIS algorithm (Python implementation) — alternative to the linear scoring method.
  2. Click — Python CLI framework — to evolve your CLI if it grows.
  3. PyYAML docs — to understand more loading options.
  4. Hypothesis — property-based testing — deeper testing of the scoring logic.
  5. Decision making with multi-objective optimization — applicable theory if you scale the tool.
← PreviousThe full comparison tableNext →Module 8: Unified AI Client — Final integrating project