6. Decision Matrix Framework

Overview

In this capsule you'll build the framework that turns trade-off analysis into a documented, defensible decision. The decision matrix isn't an academic exercise — it's the tool you produce in the module project and reuse in Modules 7 and 8. By the end, you'll have a functional template and know how to fill it in for any use case.

Context: The previous capsules gave you the inputs: deployment categories, trade-offs across 5 dimensions, serverless vs containers comparison, and cost modeling. This capsule gives you the structure to synthesize it all into a decision.

---

What a Decision Matrix Is

Definition

A decision matrix is a table that evaluates options against weighted criteria, producing a numeric score that indicates which option aligns best with your requirements. It doesn't replace judgment — it structures it.

Decision Matrix = Criteria + Weights + Evaluation + Score

Criteria:   The dimensions that matter for your case
Weights:    The relative importance of each criterion (sum to 100)
Evaluation: The score of each option on each criterion (1-5)
Score:      Weight × Evaluation, summed per option

Why it works for deployment

Without a matrix, deployment decisions are: "my friend uses Railway" or "AWS is what serious companies use." With a matrix, the decision is: "given my constraints of budget, team, and traffic, Railway has a score of 4.2 vs AWS with 3.1 — and I can explain why."

The matrix is especially valuable when:

• You have to justify the decision to a team or manager
• You need to re-evaluate when conditions change
• You want to document the reasoning for your future self

---

Anatomy of a Decision Matrix

Complete structure

# Decision Matrix: [Project name]

## Context
- Project: [Which AI app]
- Stage: [MVP / Growth / Scale]
- Team: [Size and skills]
- Budget: [Monthly range]
- Timeline: [Urgency]

## Criteria and Weights

| # | Criterion | Weight | Weight justification |
|---|----------|------|----------------------|
| 1 | Monthly cost | 25 | Limited budget, <$50/month |
| 2 | Operational complexity | 20 | Team of 1, I can't dedicate >2hrs/week |
| 3 | Time-to-deploy | 20 | I need the MVP online this week |
| 4 | Scalability | 15 | Expected growth but not urgent |
| 5 | Control | 10 | No compliance requirements |
| 6 | AI-specific (cold starts, streaming) | 10 | Chat with streaming is a key feature |
| **Total** | | **100** | |

## Evaluation (1-5, where 5 = best)

| Criterion (Weight) | Local | Serverless | Managed | Self-hosted |
|-----------------|:-----:|:----------:|:-------:|:-----------:|
| Cost (25) | 4 | 5 | 4 | 1 |
| Complexity (20) | 3 | 4 | 5 | 1 |
| Time-to-deploy (20) | 3 | 4 | 5 | 1 |
| Scalability (15) | 2 | 5 | 3 | 4 |
| Control (10) | 4 | 2 | 3 | 5 |
| AI-specific (10) | 5 | 2 | 4 | 5 |

## Weighted Scores

| Criterion | Local | Serverless | Managed | Self-hosted |
|----------|:-----:|:----------:|:-------:|:-----------:|
| Cost | 100 | 125 | 100 | 25 |
| Complexity | 60 | 80 | 100 | 20 |
| Time-to-deploy | 60 | 80 | 100 | 20 |
| Scalability | 30 | 75 | 45 | 60 |
| Control | 40 | 20 | 30 | 50 |
| AI-specific | 50 | 20 | 40 | 50 |
| **TOTAL** | **340** | **400** | **415** | **225** |

## Decision
- **Recommendation:** Managed (Railway) — Score 415
- **Second option:** Serverless (Lambda) — Score 400
- **Ruled out:** Self-hosted (Score 225, overkill for the case)

## Re-evaluation conditions
- If traffic > 500 req/hour → re-evaluate serverless vs managed
- If I need streaming → confirm that managed supports it well
- If budget > $200/month → consider local (VPS) for more control
- Scheduled review: 3 months after deploy

---

Step by Step: How to Build Your Matrix

Step 1: Define the context

Before evaluating options, document your situation. Without context, the criteria and weights are arbitrary.

context = {
    "project": "RAG app for internal documentation",
    "stage": "MVP",
    "team": "1 full-time developer",
    "budget": "$0-50/month",
    "timeline": "Online in 1 week",
    "users": "~50 internal",
    "traffic_pattern": "Business hours, 9am-6pm",
    "ai_features": ["chat", "search", "document Q&A"],
    "constraints": ["streaming for chat", "no special compliance"],
}

Step 2: Select relevant criteria

Don't use the same criteria for every project. Select the ones that matter for YOUR case:

# Standard criteria (start with these)
standard_criteria = [
    "Monthly cost",
    "Operational complexity",
    "Time-to-deploy",
    "Scalability",
    "Control",
]

# AI-specific criteria (add as needed)
ai_criteria = [
    "Cold starts (first request latency)",
    "Streaming support (SSE/WebSocket)",
    "Memory for models/embeddings",
    "GPU availability",
    "Timeout limits",
]

# For this case: standard + streaming + cold starts
selected_criteria = standard_criteria + [
    "Streaming support",
    "Cold starts",
]

Step 3: Assign weights (must sum to 100)

The weights reflect YOUR priorities, not universal "best practices":

weights = {
    "Monthly cost": 25,           # Budget is the main constraint
    "Operational complexity": 20, # Just me, I need it to be simple
    "Time-to-deploy": 20,         # I need the MVP fast
    "Scalability": 10,            # Not urgent, 50 users
    "Control": 5,                 # No compliance
    "Streaming support": 15,      # Key feature for chat
    "Cold starts": 5,             # Acceptable if <3s
}

assert sum(weights.values()) == 100, "Weights must sum to 100"

Step 4: Evaluate each option (1-5)

Be honest and specific. "5" doesn't mean "perfect" — it means "the best option on this dimension."

evaluation = {
    "Monthly cost": {
        "Local": 4,        # VPS $24/month, predictable
        "Serverless": 5,   # ~$1-5/month at this volume
        "Managed": 5,      # Free tier covers the MVP
        "Self-hosted": 1,  # $30 infra + $400 ops
    },
    "Streaming support": {
        "Local": 5,        # FastAPI + uvicorn native
        "Serverless": 1,   # Lambda doesn't support streaming
        "Managed": 4,      # Railway supports WebSockets
        "Self-hosted": 5,  # Full control
    },
    # ... other criteria
}

Step 5: Calculate weighted scores

def calculate_matrix(weights: dict, evaluation: dict) -> dict:
    """Calculate weighted scores for each option."""
    options = ["Local", "Serverless", "Managed", "Self-hosted"]
    scores = {opt: 0 for opt in options}

    for criterion, weight in weights.items():
        for option in options:
            score = evaluation[criterion][option]
            scores[option] += weight * score

    return dict(sorted(scores.items(), key=lambda x: x[1], reverse=True))

results = calculate_matrix(weights, evaluation)
# {'Managed': 415, 'Serverless': 400, 'Local': 340, 'Self-hosted': 225}

Step 6: Document the decision and re-evaluation conditions

The decision isn't only "what I chose" but "under what conditions I'll re-evaluate":

decision = {
    "recommendation": "Managed (Railway)",
    "score": 415,
    "runner_up": "Local (VPS)",
    "runner_up_score": 340,
    "reasoning": "Streaming support ruled out Serverless despite a good score. "
                 "Railway offers WebSockets + free tier + deploy in minutes.",
    "re_evaluate_when": [
        "Traffic exceeds 10K req/day",
        "I need a GPU for local models",
        "Budget changes to >$100/month",
        "Railway changes pricing or discontinues features",
    ],
    "review_date": "3 months from deploy",
}

---

Common Decision Matrix Patterns

Pattern 1: MVP with zero budget

Dominant weight: Cost (40) + Time-to-deploy (30)
Typical result: Managed (free tier) wins
Exception: If you need streaming and the platform doesn't support it → Local

Pattern 2: Growing startup

Dominant weight: Scalability (30) + Cost (25)
Typical result: Serverless or Managed with auto-scale
Exception: If the workload is constant (no peaks) → Local (VPS, fixed cost)

Pattern 3: Enterprise with compliance

Dominant weight: Control (35) + Security (25, extra criterion)
Typical result: Self-hosted or Local in a private VPC
Exception: Rarely — compliance usually requires control

Pattern 4: AI with local models

Dominant weight: Memory/GPU (30) + Control (25)
Typical result: Self-hosted (GPU instances)
Exception: If the model is small (<4GB) → Local on a VPS with enough RAM

---

Complete Example: Decision Matrix from Start to Finish

We're going to build a complete decision matrix for a concrete case. Follow each step and at the end you'll have a numeric result with a recommendation.

The case: NotifyAI

NotifyAI is a service that analyzes mentions of your brand on social media and sends a daily summary by email using an LLM.

context = {
    "project": "NotifyAI — Social media AI monitor",
    "stage": "MVP",
    "team": "1 developer (you)",
    "budget": "$0-25/month",
    "users": "Just you + 3 beta clients",
    "traffic": "~200 requests/day (mention analysis)",
    "ai_features": ["Sentiment analysis", "Summary generation"],
    "constraints": ["No streaming", "Latency not critical (batch process)"],
}

Step by step with NotifyAI

# Step 1: Criteria and weights (sum to 100)
criteria = {
    "Monthly cost":          35,  # Budget is constraint #1
    "Operational complexity": 25,  # Just me, I need something simple
    "Time-to-deploy":        25,  # I want to launch this week
    "Scalability":            5,  # 200 req/day, I don't need scale
    "Control":               10,  # No compliance, but I want easy debug
}

# Step 2: Evaluation (1-5) with justification
evaluation = {
    "Monthly cost":           {"Local": 4, "Serverless": 5, "Managed": 5, "Self-hosted": 1},
    "Operational complexity": {"Local": 3, "Serverless": 4, "Managed": 5, "Self-hosted": 1},
    "Time-to-deploy":         {"Local": 3, "Serverless": 3, "Managed": 5, "Self-hosted": 1},
    "Scalability":            {"Local": 2, "Serverless": 5, "Managed": 3, "Self-hosted": 4},
    "Control":                {"Local": 4, "Serverless": 2, "Managed": 3, "Self-hosted": 5},
}

# Step 3: Calculation
options = ["Local", "Serverless", "Managed", "Self-hosted"]
totals = {opt: 0 for opt in options}

print(f"{'Criterion':<25} {'Weight':>4} | {'Local':>6} {'Server':>7} {'Managed':>8} {'Self':>6}")
print("-" * 70)

for criterion, weight in criteria.items():
    scores = evaluation[criterion]
    row = f"{criterion:<25} {weight:>4} |"
    for opt in options:
        weighted = weight * scores[opt]
        totals[opt] += weighted
        row += f" {weighted:>6}"
    print(row)

print("-" * 70)
row = f"{'TOTAL':<25} {'':>4} |"
for opt in options:
    row += f" {totals[opt]:>6}"
print(row)

Output:

Criterion                 Weight | Local  Server  Managed   Self
----------------------------------------------------------------------
Monthly cost                  35 |    140    175      175     35
Operational complexity        25 |     75    100      125     25
Time-to-deploy                25 |     75     75      125     25
Scalability                    5 |     10     25       15     20
Control                       10 |     40     20       30     50
----------------------------------------------------------------------
TOTAL                            |    340    395      470    155

Decision: Managed (470) wins by a clear margin. For an MVP of 200 req/day with 1 developer and a minimal budget, Railway with a free tier is the optimal option. Serverless (395) is a viable alternative. Self-hosted (155) is out of the discussion.

Reusable script: decision_matrix.py

# decision_matrix.py — Copy, modify the data, run

def run_decision_matrix(criteria: dict, evaluation: dict):
    """Run a complete decision matrix and show results."""
    options = list(list(evaluation.values())[0].keys())

    assert sum(criteria.values()) == 100, f"Weights sum to {sum(criteria.values())}, must sum to 100"

    totals = {opt: 0 for opt in options}
    for criterion, weight in criteria.items():
        for opt in options:
            totals[opt] += weight * evaluation[criterion][opt]

    max_possible = sum(criteria.values()) * 5
    sorted_results = sorted(totals.items(), key=lambda x: x[1], reverse=True)

    print(f"\n{'='*50}")
    print(f"  DECISION MATRIX RESULTS")
    print(f"{'='*50}")
    for rank, (opt, score) in enumerate(sorted_results, 1):
        pct = score / max_possible * 100
        bar = "#" * int(pct / 5)
        marker = " ← RECOMMENDED" if rank == 1 else ""
        print(f"  {rank}. {opt:<15} {score:>4}/{max_possible}  ({pct:.0f}%)  {bar}{marker}")
    print(f"{'='*50}\n")

    winner, w_score = sorted_results[0]
    runner, r_score = sorted_results[1]
    gap = (w_score - r_score) / max_possible * 100
    if gap < 5:
        print(f"  ⚠️  Close result ({gap:.1f}% difference). Consider qualitative factors.")
    else:
        print(f"  ✅ Clear result ({gap:.1f}% difference). {winner} is the recommendation.")

    return sorted_results


# === USE YOUR DATA HERE ===
my_criteria = {
    "Monthly cost": 30,
    "Complexity": 25,
    "Time-to-deploy": 20,
    "Scalability": 15,
    "Control": 10,
}

my_evaluation = {
    "Monthly cost":   {"Local": 4, "Serverless": 5, "Managed": 4, "Self-hosted": 1},
    "Complexity":     {"Local": 3, "Serverless": 4, "Managed": 5, "Self-hosted": 1},
    "Time-to-deploy": {"Local": 3, "Serverless": 4, "Managed": 5, "Self-hosted": 1},
    "Scalability":    {"Local": 2, "Serverless": 5, "Managed": 3, "Self-hosted": 4},
    "Control":        {"Local": 4, "Serverless": 2, "Managed": 3, "Self-hosted": 5},
}

run_decision_matrix(my_criteria, my_evaluation)

---

When to Re-evaluate Your Decision Matrix

Re-evaluation triggers

A decision matrix isn't a static document. There are clear signs that you need to evaluate it again:

Trigger	Example	Action
Stage change	You go from MVP to Growth (>1K users)	Re-evaluate scalability and cost
Budget change	Your company approves $500/month for infra	Re-evaluate options previously ruled out for cost
New technical requirement	You need streaming you didn't have before	Re-evaluate: Lambda gets ruled out
Team change	You hire a DevOps	Self-hosted becomes viable
Provider pricing change	Railway raises prices or eliminates the free tier	Re-evaluate alternatives
Repeated incidents	3+ outages in a month with your current strategy	Evaluate migration

Recommended frequency

• Scheduled review: Every 3 months
• Trigger-based review: Immediately when a trigger occurs
• Light review: Monthly, verify that real costs match the estimates

What to document in each re-evaluation

## Re-evaluation: [Date]
- Trigger: [What motivated the review]
- Current data: [Real traffic, real costs, incidents]
- Did the ranking change? [Yes/No]
- Action: [Keep / Migrate to X / Investigate X]

---

Common Errors in Decision Matrices

Error 1: Uniform weights

❌ All criteria weigh 20 (5 × 20 = 100)
   → All dimensions "matter equally"
   → Result: The most "balanced" option always wins (managed)
   → Problem: It doesn't reflect YOUR priorities

✅ Differentiated weights according to YOUR case
   → If cost is your constraint, give it 30-40
   → If control is irrelevant, give it 5
   → The result reflects YOUR reality

Error 2: Evaluating options without data

❌ "Serverless: Scalability = 5" (because "Lambda scales")
   → Without verifying if it scales well for YOUR workload

✅ "Serverless: Scalability = 4"
   → Lambda scales automatically, but 8s cold starts
     with my dependencies reduce the usefulness of auto-scale
     for interactive requests

Error 3: Not documenting re-evaluation conditions

❌ Decision: "We use Railway"
   → When do we stop using it? If we grow? If prices change?

✅ Decision: "We use Railway"
   Re-evaluate when:
   - Traffic > 500 req/hour (check limits)
   - Budget > $100/month (consider a VPS)
   - We need AWS services (S3, SageMaker)
   Review: Q2 2026

---

Troubleshooting

Problem 1: "My weights feel arbitrary"

Solution: Use the pairwise comparison technique. For each pair of criteria, ask yourself: "Would I rather optimize A or B?" The criterion that wins more comparisons gets more weight. For 5 criteria, there are 10 comparisons — in 5 minutes you have more robust weights than "by eye."

Problem 2: "The winner has a functional deal-breaker"

Solution: Deal-breakers aren't captured in numeric scores. Before calculating, identify deal-breakers: if you need streaming, Lambda has score 0 (not 1-5) on that dimension. Eliminate options with deal-breakers before the numeric evaluation.

Problem 3: "My matrix gives a technical tie"

Solution: A tie (<5% difference) means both options are viable for your case. Choose by: (1) team familiarity, (2) ease of future migration, or (3) the one with better documentation. Document that it was a tie and why you chose the one you chose.

Problem 4: "I don't have real data to evaluate"

Solution: Use conservative estimates (reasonable worst case) and mark the evaluations as "estimated." After 2-4 weeks in production, update with real data. The first matrix is an informed hypothesis — it doesn't need to be perfect to be useful.

---

Hands-On Exercises

Exercise 1: Build your matrix from scratch

Use this capsule's template to build a decision matrix for your AI app. Include context, weighted criteria, 1-5 evaluation, and weighted score.

See solution

There's no fixed solution — it depends on your case. Verify that:

• The weights sum to 100
• Each evaluation has justification (not just numbers)
• There are documented re-evaluation conditions
• The decision includes a second option and ruled-out ones with a reason

If your matrix produces a clear winner (>20% over the second option), your case has strong constraints. If it's close (<10% difference), any of the top-2 is viable — choose by experience or preference.

Exercise 2: Sensitivity analysis

Take your matrix from Exercise 1. Change the weight of criterion #1 from its current value to 10 less. Does the winner change?

See solution

If changing a weight ±10 points changes the winner, your decision is sensitive to that criterion. That means:

• You need to be very precise in evaluating that criterion
• A change in your circumstances (e.g., more budget) can change the recommendation
• Document this in the re-evaluation conditions

If the winner doesn't change, your decision is robust — the option wins under multiple weight combinations.

Exercise 3: Matrix for a different case than yours

Build a matrix for a case opposite to yours: if your case is an MVP with 1 developer, do a matrix for an enterprise with a team of 10. Do the criteria change? The weights?

See solution

In an enterprise case with a team of 10:

• Control weighs more (compliance, auditing)
• Cost weighs less (larger budget)
• Scalability weighs more (more users)
• Time-to-deploy weighs less (more formal process)
• New criteria: SLA, multi-region, disaster recovery

The result probably favors Self-hosted or AWS managed services instead of Railway/Render. This exercise shows that the matrix is a framework, not a fixed answer.

Exercise 4: Pitch the decision

Prepare a 2-minute presentation of your decision matrix for a non-technical stakeholder. Include: the context, the 3 most important criteria, the winner, and when to re-evaluate.

See solution

Example:

"We need to deploy our documentation AI app. I evaluated 4 infrastructure options using 7 criteria weighted by our priorities: limited budget, a one-person team, and the need for streaming in the chat.

Railway won with 415 points out of 500 possible. The two main reasons: (1) the free tier covers our current volume, and (2) it supports WebSockets that we need for streaming. The alternative is our own server with Docker, which gives us more control but costs more in my time.

We re-evaluate in 3 months or if we exceed 500 requests/hour. I have the matrix documented for when we need it."

Key: Simplify for non-technical people. "4 options, 7 criteria, Railway wins on cost and features."

---

Summary

• A decision matrix turns trade-off analysis into a documented, defensible decision.
• The 6 steps are: context → criteria → weights → evaluation → scores → decision + re-evaluation.
• The weights must reflect YOUR priorities, not generic best practices. They must sum to 100.
• The evaluation (1-5) must have justification, not just numbers.
• The most common errors are: uniform weights, evaluation without data, and not documenting re-evaluation conditions.
• A good matrix includes sensitivity analysis: does the result change if I change a weight?
• The matrix is reused: Module 7 (platforms), Module 8 (capstone project).

---

Additional Resources

1. Decision Matrix Analysis — MindTools — General framework for decision matrices
2. Pugh Matrix — Formal weighted decision method
3. Architecture Decision Records (ADR) — Format for documenting architecture decisions
4. DACI Framework — Framework for team decisions
5. AWS Well-Architected Tool — Architecture evaluation tool
6. Technology Radar — ThoughtWorks — Reference for evaluating technologies

---