Module 7: Performance vs Cost Trade-offs

Project: Trade-off Decision Matrix

Next →

Overview

We close the module by applying the Decision Matrix template (M7-07) to a concrete AI system. The output: a professional document with 4-5 technical decisions evaluated using the framework. It's the deliverable a Tech Lead presents to their team or stakeholders.

This document is portfolio-worthy: it demonstrates that you think like a senior architect, not like a junior who decides by preference.

By the end of the project you'll have:

  • A complete Decision Matrix Document with 4-5 decisions
  • Each decision uses the complete template (criteria, scores, trade-offs, reconsider)
  • A document ready for review by peers or stakeholders

The case to evaluate

System: an AI-Powered Customer Support for a B2B SaaS (similar to M5-08, expanded).

Context:

  • 5 pilot companies (50-200 employees each)
  • Expecting 50 companies in 12 months
  • Per company: ~200 queries/day average, peaks at 600/day
  • Knowledge base per company: ~10K documents, ~3M tokens
  • SLA: P95 <8s, 99.5% availability
  • Operational budget: $5,000/month initial, scalable with revenue

Technical decisions to evaluate:

  1. LLM Provider (single vs multi-provider)
  2. Vector Database (managed vs self-hosted)
  3. Hosting (serverless vs containers vs hybrid)
  4. Cache strategy (yes vs no, what level)
  5. Auth & Multi-tenant approach

Your deliverable evaluates at least 4 of the 5, each one with a complete Decision Matrix.

Structure of the final document

# Architecture Decisions — AI Customer Support v1.0

## Summary
This document contains the key technical decisions for the AI Customer
Support system. Each decision uses the Decision Matrix framework with
quantifiable criteria and documented trade-offs.

Decisions covered:
1. LLM Provider strategy
2. Vector Database choice
3. Hosting paradigm
4. Cache strategy
5. Multi-tenant architecture

Total estimated monthly cost (Year 1): ~$2,750/mo
Total estimated monthly cost (Year 2 with growth): ~$8,500/mo

---

## Decision 1: LLM Provider Strategy

[Full Decision Matrix using the M7-07 template]

---

## Decision 2: Vector Database

[Full Decision Matrix]

---

## Decision 3: Hosting Paradigm

[Full Decision Matrix]

---

## Decision 4: Cache Strategy

[Full Decision Matrix]

---

## Cross-decision interactions

[A section showing how the decisions interact with each other]

---

## Total cost projection

[Table with projected cost per component, Year 1 and Year 2]

---

## Risks and mitigations

[3-5 identified risks with mitigation plans]

Worked example: Decision 1 (LLM Provider Strategy)

This is to calibrate the level of detail:

## Decision 1: LLM Provider Strategy

**Status**: Accepted
**Date**: 2026-05-11
**Decider(s)**: Mike Nieva (Tech Lead), CTO

### Context
Need to choose LLM provider strategy for the customer support assistant.
Expected volume: 1.2M queries/mo by end of Year 1, growing to 5M by Year 2.
Mix of: ~60% factual lookups (low complexity), ~30% explanations (medium),
~10% complex reasoning.

Tested against eval set of 100 representative queries from pilot data.

### Constraints
- Budget: $1,500-2,500/mo for LLM (within total $5K budget)
- Compliance: SOC2 (clients), HIPAA capability for 2 prospects
- Latency: P95 <8s
- Multi-language: Spanish + English

### Options considered

**Option A: OpenAI single provider with routing**
gpt-4o-mini for simple/medium, gpt-4o for complex queries. Single integration,
familiar to team.

**Option B: Multi-provider with primary/fallback**
OpenAI primary + Anthropic Claude as fallback. Circuit breaker switches on
degradation.

**Option C: Self-hosted Mistral 8B + OpenAI for complex**
Hybrid: Mistral hosted on Modal for simple, OpenAI for complex queries.

**Option D: OpenAI for all (no routing)**
Default to gpt-4o-mini, no model routing. Simplest.

### Evaluation criteria

| Criterion | Weight | Anchor |
|-----------|--------|--------|
| Cost @ 1.2M queries/mo | 0.25 | 5=<$1K, 4=$1-1.5K, 3=$1.5-2.5K, 2=$2.5-4K, 1=>$4K |
| Quality (eval pass rate) | 0.25 | 5=>90%, 4=85-90%, 3=80-85%, 2=75-80%, 1=<75% |
| Time to market | 0.20 | 5=1wk, 3=2-4wk, 1=6wk+ |
| Vendor risk | 0.15 | 5=multi-provider redundancy, 1=single point fail |
| Operational complexity | 0.10 | 5=zero ops, 1=high ops |
| Compliance | 0.05 | 5=full, 1=none |

### Evaluation matrix

| Criterion | Weight | A: OpenAI Routing | B: Multi-provider | C: Hybrid SH | D: OpenAI single |
|-----------|--------|-------------------|--------------------|--------------|------------------|
| Cost | 0.25 | 4 ($1.2K) | 3 ($1.5K) | 3 ($1.4K) | 3 ($1.6K) |
| Quality | 0.25 | 5 (89%) | 5 (89%) | 3 (82%) | 5 (89%) |
| Time to market | 0.20 | 4 (2wk) | 3 (4wk) | 2 (5wk) | 5 (1wk) |
| Vendor risk | 0.15 | 2 (single) | 5 (dual) | 4 (mix) | 2 (single) |
| Operational | 0.10 | 4 | 4 | 2 | 5 |
| Compliance | 0.05 | 5 | 5 | 4 | 5 |
| **Weighted** | | **4.00** | **4.00** | **2.90** | **4.05** |

### Decision: **Option A (OpenAI Routing)**

Option D scores marginally higher (4.05 vs 4.00), but we choose A: routing's
cost efficiency compounds at scale, a strategic factor the point-in-time matrix
underweights. The 30% complex queries justify gpt-4o; the 70% simple don't need
it — and that gap widens toward 5M queries/mo. A close call decided by a
qualitative factor (see M7-07).

### Rationale
- Quality threshold met (89% in eval, >85% target)
- Best cost-quality balance via routing
- Familiar to team, 2-week implementation
- Vendor risk accepted: OpenAI mature, multi-provider can be added Q2

### Trade-offs accepted
- ✅ Significantly lower cost than "all gpt-4o" (would be ~$5K/mo)
- ✅ Quality maintained on complex queries (escalate to gpt-4o)
- ⚠️ Single vendor risk: if OpenAI has outage, we degrade
- ⚠️ Vendor lock-in: 4-6 weeks to migrate to Anthropic if needed
- ⚠️ Cost grows with volume: at 5M queries/mo, ~$5K/mo

### Reconsider if
1. Volume exceeds 3M queries/mo (cost optimization needed; add multi-provider)
2. OpenAI has prolonged outage (>4hrs in 30 days)
3. Client demands HIPAA on-prem (would need Self-hosted EU)
4. Anthropic releases model significantly better at lower cost
5. Cost grows beyond $3K/mo for LLM alone

### References
- Eval set results: `s3://eval-results/2026-05-11-llm-comparison.json`
- OpenAI pricing: https://openai.com/pricing (snapshot 2026-05-11)
- Internal benchmark: Vector Quality vs Cost across providers

How to work through this project

I suggest this order:

  1. Skim the case (10 min): familiarize yourself with the system and the constraints
  2. Decision 1: LLM Provider (1 hour): following the example above
  3. Decision 2: Vector DB (45 min): apply the M7-04 framework
  4. Decision 3: Hosting (45 min): apply M7-05
  5. Decision 4: Cache (45 min): apply M7-03
  6. Decision 5 (optional): Multi-tenant (45 min)
  7. Cross-decision interactions (30 min): describe how the decisions affect each other
  8. Cost projection (15 min): a table with Year 1 and Year 2 totals
  9. Risks (15 min): top 3-5 with mitigations

Total: ~5-6 hours for the complete document.

Evaluation criteria

To ensure your document is production-ready:

  • Minimum 4 decisions covered with a complete Decision Matrix
  • Each Decision Matrix has:
    • Specific context and constraints
    • Minimum 3 options considered
    • Criteria with quantitative anchors (not vague)
    • A matrix with scores and a weighted total
    • A clear rationale
    • Explicit trade-offs (not just positives)
    • Reconsider triggers (minimum 3)
  • Cross-decision interactions documented (not isolated decisions)
  • Cost projection numerical for Year 1 and Year 2
  • Risks identified with mitigations
  • The document is presentable to non-technical stakeholders (executive summary)

Cross-decision interactions

This section is what differentiates a Tech Lead from a junior. The decisions interact:

## Cross-decision interactions

### Decision 1 (LLM Provider) ↔ Decision 4 (Cache strategy)
The routing strategy depends on the cache. If we cache gpt-4o-mini responses,
it reduces the $1.2K monthly by ~35%. If we cache gpt-4o responses (complex
queries), the savings are disproportionate (gpt-4o costs 10× more). Decision:
prioritize caching queries that go to gpt-4o.

### Decision 2 (Vector DB) ↔ Decision 3 (Hosting)
If we choose Pinecone managed (Decision 2), our hosting doesn't need to
handle vector DB infra. If self-hosted ChromaDB, hosting needs persistent
volumes + more memory → it changes the hosting options.

### Decision 5 (Multi-tenant) ↔ Decision 1 (LLM Provider)
If we support HIPAA per tenant, that tenant can't use standard OpenAI
(requires a BAA). We'd need Azure OpenAI or self-hosted Mistral. This adds
complexity to routing: besides by query type, route by tenant compliance.

By showing these interactions you prove you thought about the system as a coherent whole, not isolated decisions.

Complete cost projection

## Total cost projection

### Year 1 (1.2M queries/mo at end)

| Component | Monthly Cost |
|-----------|--------------|
| LLM (OpenAI routing) | $1,200 |
| Vector DB (Pinecone) | $400 |
| Hosting (Cloud Run + workers) | $700 |
| Cache (Redis cluster) | $200 |
| Monitoring (Datadog basic) | $150 |
| Other (SES, S3, etc) | $100 |
| **Total Y1** | **$2,750/mo** |

### Year 2 (5M queries/mo)

| Component | Monthly Cost | Change | Note |
|-----------|--------------|--------|------|
| LLM | $5,000 | +$3,800 | Scales linearly |
| Vector DB | $1,200 | +$800 | More vectors |
| Hosting | $1,500 | +$800 | More instances |
| Cache | $300 | +$100 | Bigger cluster |
| Monitoring | $300 | +$150 | More data |
| Other | $200 | +$100 | |
| **Total Y2** | **$8,500/mo** | | |

### Cost per query analysis

- Y1: $2,750 / 1,200,000 = **$0.0023 per query**
- Y2: $8,500 / 5,000,000 = **$0.0017 per query** (improves with scale)

### Optimization opportunities Y2+
1. Migrate to multi-provider (Decision 1 reconsider): potential $1-2K savings
2. Self-hosted Pinecone alternative if volume justifies: $400-600 savings
3. Increase cache hit rate from 35% to 50%: $1K savings on LLM

Typical risks to include

## Risks identified

### Risk 1: OpenAI outage / pricing change
- **Likelihood**: medium (historical: 2-3 minor outages/year)
- **Impact**: high (service degradation for all clients)
- **Mitigation**:
  - Multi-provider fallback (Anthropic) implemented Q2 Y1
  - Circuit breaker auto-switches on degradation
  - Status page transparency with clients

### Risk 2: Vector DB performance under high load
- **Likelihood**: medium
- **Impact**: medium (latency degradation, not failure)
- **Mitigation**:
  - Load test before each major release
  - Pre-emptive scale at 70% capacity threshold
  - Consider self-hosted backup at $400/mo (Decision 2 reconsider)

### Risk 3: Cost growth exceeds revenue
- **Likelihood**: high (typical SaaS scaling)
- **Impact**: medium (margin pressure)
- **Mitigation**:
  - Quarterly cost-per-customer review
  - Routing rules adjustment based on observed query patterns
  - Cache hit rate optimization (current: 35%, target: 50% Y1 end)

### Risk 4: Compliance gaps with new client
- **Likelihood**: medium (HIPAA prospect already in pipeline)
- **Impact**: high (loses deal or requires major rework)
- **Mitigation**:
  - Architecture allows per-tenant LLM routing
  - Modal EU + self-hosted Mistral as compliant option
  - Pre-sales technical review by Tech Lead

Connection with M8 (Capstone)

This document is NOT discarded. It's a key section of the Capstone Architecture Design in M8. When you reach M8, you'll include the Decision Matrix as part of the full document.

If you worked well here, M8 becomes "integrate everything + add C4 diagrams".

Evidence of success by the end of M7

You'll know you finished well if:

  • ✅ The document contains 4-5 complete Decision Matrices with quantifiable anchors
  • ✅ Each decision has explicit trade-offs and reconsider triggers
  • ✅ Cross-decision interactions are documented
  • ✅ The cost projection is numerical and defensible
  • ✅ Risks identified with specific mitigations
  • ✅ A peer can review and give constructive feedback based on data, not opinions

Module 7 completed

You went from making decisions by intuition to making them by framework. Specifically:

  • ✅ Five specific trade-offs (scale, cache, managed, serverless, routing)
  • ✅ A reusable Decision Matrix template
  • ✅ A project applied to a concrete system

What comes in M8 is integrating the whole path (M1-M7) into the final Capstone Architecture Design.

Next module

Module 8 — Capstone Architecture Design. The final capstone project. You take everything learned in M1-M7 and produce the complete blueprint of the AI-Powered Knowledge Assistant. It's the most portfolio-worthy document in the guide.

Resources

  1. ADR examples by industry — professional examples.
  2. Google SRE — Architecture decision documentation — documentation patterns.
  3. Mozilla's architecture decision record template.
  4. DACI framework — for group decisions.
  5. Stripe's tech radar — public examples of tech decisions.
← PreviousDecision Matrix templateNext →Introduction to Capstone Architecture Design