Módulo 7: Performance vs Cost Trade-offs

Proyecto: Trade-off Decision Matrix

Siguiente →

Descripción

Cerramos el módulo aplicando la Decision Matrix template (M7-07) a un sistema AI concreto. El output: un documento profesional con 4-5 decisiones técnicas evaluadas usando el framework. Es el deliverable que un Tech Lead presenta a su equipo o stakeholders.

Este documento es portfolio-worthy: demuestra que pensás como senior architect, no como junior que decide por preference.

Al terminar el proyecto vas a tener:

  • Decision Matrix Document completo con 4-5 decisiones
  • Cada decisión usa el template completo (criteria, scores, trade-offs, reconsider)
  • Documento ready para revisión por peers o por stakeholders

El caso a evaluar

Sistema: AI-Powered Customer Support para una SaaS B2B (similar al de M5-08, expanded).

Contexto:

  • 5 empresas pilot (50-200 empleados cada una)
  • Esperando 50 empresas en 12 meses
  • Por empresa: ~200 queries/día promedio, picos a 600/día
  • Knowledge base por empresa: ~10K documentos, ~3M tokens
  • SLA: P95 <8s, disponibilidad 99.5%
  • Budget operacional: $5,000/mes inicial, escalable según ingresos

Decisiones técnicas a evaluar:

  1. LLM Provider (single vs multi-provider)
  2. Vector Database (managed vs self-hosted)
  3. Hosting (serverless vs containers vs híbrido)
  4. Cache strategy (sí vs no, qué nivel)
  5. Auth & Multi-tenant approach

Tu deliverable evalúa al menos 4 de las 5, cada una con un Decision Matrix completo.

Estructura del documento final

# Architecture Decisions — AI Customer Support v1.0

## Summary
Este documento contiene las decisiones técnicas clave para el sistema
AI Customer Support. Cada decisión usa Decision Matrix framework con
criteria cuantificables y trade-offs documentados.

Decisiones cubiertas:
1. LLM Provider strategy
2. Vector Database choice
3. Hosting paradigm
4. Cache strategy
5. Multi-tenant architecture

Total estimated monthly cost (Year 1): ~$3,200/mo
Total estimated monthly cost (Year 2 with growth): ~$8,000/mo

---

## Decision 1: LLM Provider Strategy

[Full Decision Matrix usando el template de M7-07]

---

## Decision 2: Vector Database

[Full Decision Matrix]

---

## Decision 3: Hosting Paradigm

[Full Decision Matrix]

---

## Decision 4: Cache Strategy

[Full Decision Matrix]

---

## Cross-decision interactions

[Sección que muestra cómo las decisiones interactúan entre sí]

---

## Total cost projection

[Tabla con costo proyectado por componente, Year 1 y Year 2]

---

## Risks and mitigations

[3-5 riesgos identificados con planes de mitigación]

Ejemplo trabajado: Decisión 1 (LLM Provider Strategy)

Esto es para calibrar nivel de detalle:

## Decision 1: LLM Provider Strategy

**Status**: Accepted
**Date**: 2026-05-11
**Decider(s)**: Mike Nieva (Tech Lead), CTO

### Context
Need to choose LLM provider strategy for the customer support assistant.
Expected volume: 1.2M queries/mo by end of Year 1, growing to 5M by Year 2.
Mix of: ~60% factual lookups (low complexity), ~30% explanations (medium),
~10% complex reasoning.

Tested against eval set of 100 representative queries from pilot data.

### Constraints
- Budget: $1,500-2,500/mo for LLM (within total $5K budget)
- Compliance: SOC2 (clients), HIPAA capability for 2 prospects
- Latency: P95 <8s
- Multi-language: Spanish + English

### Options considered

**Option A: OpenAI single provider with routing**
gpt-4o-mini for simple/medium, gpt-4o for complex queries. Single integration,
familiar to team.

**Option B: Multi-provider with primary/fallback**
OpenAI primary + Anthropic Claude as fallback. Circuit breaker switches on
degradation.

**Option C: Self-hosted Mistral 8B + OpenAI for complex**
Hybrid: Mistral hosted on Modal for simple, OpenAI for complex queries.

**Option D: OpenAI for all (no routing)**
Default to gpt-4o-mini, no model routing. Simplest.

### Evaluation criteria

| Criterion | Weight | Anchor |
|-----------|--------|--------|
| Cost @ 1.2M queries/mo | 0.25 | 5=<$1K, 4=$1-1.5K, 3=$1.5-2.5K, 2=$2.5-4K, 1=>$4K |
| Quality (eval pass rate) | 0.25 | 5=>90%, 4=85-90%, 3=80-85%, 2=75-80%, 1=<75% |
| Time to market | 0.20 | 5=1wk, 3=2-4wk, 1=6wk+ |
| Vendor risk | 0.15 | 5=multi-provider redundancy, 1=single point fail |
| Operational complexity | 0.10 | 5=zero ops, 1=high ops |
| Compliance | 0.05 | 5=full, 1=none |

### Evaluation matrix

| Criterion | Weight | A: OpenAI Routing | B: Multi-provider | C: Hybrid SH | D: OpenAI single |
|-----------|--------|-------------------|--------------------|--------------|------------------|
| Cost | 0.25 | 4 ($1.2K) | 3 ($1.5K) | 3 ($1.4K) | 3 ($1.6K) |
| Quality | 0.25 | 5 (89%) | 5 (89%) | 3 (82%) | 5 (89%) |
| Time to market | 0.20 | 4 (2wk) | 3 (4wk) | 2 (5wk) | 5 (1wk) |
| Vendor risk | 0.15 | 2 (single) | 5 (dual) | 4 (mix) | 2 (single) |
| Operational | 0.10 | 4 | 4 | 2 | 5 |
| Compliance | 0.05 | 5 | 5 | 4 | 5 |
| **Weighted** | | **3.95** | **3.85** | **2.85** | **3.95** |

### Decision: **Option A (OpenAI Routing)**

Tied with Option D, but A wins on cost efficiency thanks to routing. The
30% complex queries justify gpt-4o; the 70% simple don't need it.

### Rationale
- Quality threshold met (89% in eval, >85% target)
- Best cost-quality balance via routing
- Familiar to team, 2-week implementation
- Vendor risk accepted: OpenAI mature, multi-provider can be added Q2

### Trade-offs accepted
- ✅ Significantly lower cost than "all gpt-4o" (would be ~$5K/mo)
- ✅ Quality maintained on complex queries (escalate to gpt-4o)
- ⚠️ Single vendor risk: if OpenAI has outage, we degrade
- ⚠️ Vendor lock-in: 4-6 weeks to migrate to Anthropic if needed
- ⚠️ Cost grows with volume: at 5M queries/mo, ~$5K/mo

### Reconsider if
1. Volume exceeds 3M queries/mo (cost optimization needed; add multi-provider)
2. OpenAI has prolonged outage (>4hrs in 30 days)
3. Client demands HIPAA on-prem (would need Self-hosted EU)
4. Anthropic releases model significantly better at lower cost
5. Cost grows beyond $3K/mo for LLM alone

### References
- Eval set results: `s3://eval-results/2026-05-11-llm-comparison.json`
- OpenAI pricing: https://openai.com/pricing (snapshot 2026-05-11)
- Internal benchmark: Vector Quality vs Cost across providers

Cómo trabajar este proyecto

Sugiero este orden:

  1. Skim el caso (10 min): familiarizate con el sistema y las constraints
  2. Decision 1: LLM Provider (1 hora): siguiendo el ejemplo arriba
  3. Decision 2: Vector DB (45 min): aplicá el framework de M7-04
  4. Decision 3: Hosting (45 min): aplicá M7-05
  5. Decision 4: Cache (45 min): aplicá M7-03
  6. Decision 5 (opcional): Multi-tenant (45 min)
  7. Cross-decision interactions (30 min): describí cómo las decisiones se afectan
  8. Cost projection (15 min): tabla con totales Year 1 y Year 2
  9. Risks (15 min): top 3-5 con mitigations

Total: ~5-6 horas para documento completo.

Criterios de evaluación

Para asegurar tu document es production-ready:

  • Mínimo 4 decisiones cubiertas con Decision Matrix completa
  • Cada Decision Matrix tiene:
    • Context y constraints específicos
    • Mínimo 3 options considered
    • Criteria con anchors cuantitativos (no vagos)
    • Matrix con scores y weighted total
    • Rationale clear
    • Trade-offs explícitos (no solo positives)
    • Reconsider triggers (mínimo 3)
  • Cross-decision interactions documentadas (no decisiones aisladas)
  • Cost projection numérica para Year 1 y Year 2
  • Risks identified con mitigations
  • Documento es presentable a stakeholders no-técnicos (executive summary)

Cross-decision interactions

Esta sección es lo que diferencia un Tech Lead de un junior. Las decisiones interactúan:

## Cross-decision interactions

### Decision 1 (LLM Provider) ↔ Decision 4 (Cache strategy)
La estrategia de routing depende del cache. Si cacheamos respuestas de
gpt-4o-mini, reduce los $1.2K mensuales en ~35%. Si cacheamos respuestas
de gpt-4o (complex queries), el ahorro es desproporcional (gpt-4o cuesta
10× más). Decision: priorizar cache de queries que van a gpt-4o.

### Decision 2 (Vector DB) ↔ Decision 3 (Hosting)
Si elegimos Pinecone managed (Decision 2), nuestro hosting no necesita
manejar vector DB infra. Si self-hosted ChromaDB, hosting necesita
volumes persistentes + más memoria → cambia las opciones de hosting.

### Decision 5 (Multi-tenant) ↔ Decision 1 (LLM Provider)
Si soportamos HIPAA por tenant, ese tenant no puede usar OpenAI estándar
(requiere BAA). Necesitaríamos Azure OpenAI o self-hosted Mistral. Esto
agrega complexity al routing: además de por query type, routear por
tenant compliance.

Mostrando estas interacciones probás que pensaste en el sistema como conjunto coherente, no decisiones aisladas.

Cost projection completa

## Total cost projection

### Year 1 (1.2M queries/mo at end)

| Component | Monthly Cost |
|-----------|--------------|
| LLM (OpenAI routing) | $1,200 |
| Vector DB (Pinecone) | $400 |
| Hosting (Cloud Run + workers) | $700 |
| Cache (Redis cluster) | $200 |
| Monitoring (Datadog basic) | $150 |
| Other (SES, S3, etc) | $100 |
| **Total Y1** | **$2,750/mo** |

### Year 2 (5M queries/mo)

| Component | Monthly Cost | Change | Note |
|-----------|--------------|--------|------|
| LLM | $5,000 | +$3,800 | Scales linearly |
| Vector DB | $1,200 | +$800 | More vectors |
| Hosting | $1,500 | +$800 | More instances |
| Cache | $300 | +$100 | Bigger cluster |
| Monitoring | $300 | +$150 | More data |
| Other | $200 | +$100 | |
| **Total Y2** | **$8,500/mo** | | |

### Cost per query analysis

- Y1: $2,750 / 1,200,000 = **$0.0023 per query**
- Y2: $8,500 / 5,000,000 = **$0.0017 per query** (improves with scale)

### Optimization opportunities Y2+
1. Migrate to multi-provider (Decision 1 reconsider): potential $1-2K savings
2. Self-hosted Pinecone alternative if volume justifies: $400-600 savings
3. Increase cache hit rate from 35% to 50%: $1K savings on LLM

Riesgos típicos a incluir

## Risks identified

### Risk 1: OpenAI outage / pricing change
- **Likelihood**: medium (historical: 2-3 minor outages/year)
- **Impact**: high (service degradation for all clients)
- **Mitigation**:
  - Multi-provider fallback (Anthropic) implemented Q2 Y1
  - Circuit breaker auto-switches on degradation
  - Status page transparency with clients

### Risk 2: Vector DB performance under high load
- **Likelihood**: medium
- **Impact**: medium (latency degradation, not failure)
- **Mitigation**:
  - Load test before each major release
  - Pre-emptive scale at 70% capacity threshold
  - Consider self-hosted backup at $400/mo (Decision 2 reconsider)

### Risk 3: Cost growth exceeds revenue
- **Likelihood**: high (typical SaaS scaling)
- **Impact**: medium (margin pressure)
- **Mitigation**:
  - Quarterly cost-per-customer review
  - Routing rules adjustment based on observed query patterns
  - Cache hit rate optimization (current: 35%, target: 50% Y1 end)

### Risk 4: Compliance gaps with new client
- **Likelihood**: medium (HIPAA prospect already in pipeline)
- **Impact**: high (loses deal or requires major rework)
- **Mitigation**:
  - Architecture allows per-tenant LLM routing
  - Modal EU + self-hosted Mistral as compliant option
  - Pre-sales technical review by Tech Lead

Conexión con M8 (Capstone)

Este documento NO se descarta. Es una sección clave del Capstone Architecture Design en M8. Cuando llegues a M8, vas a incluir el Decision Matrix como parte del documento full.

Si trabajaste bien acá, M8 se vuelve "integrar todo + agregar diagramas C4".

Evidencia de éxito al terminar M7

Vas a saber que terminaste bien si:

  • ✅ Documento contiene 4-5 Decision Matrices completas con anclas cuantificables
  • ✅ Cada decisión tiene trade-offs explícitos y reconsider triggers
  • ✅ Cross-decision interactions están documentadas
  • ✅ Cost projection es numérica y defendible
  • ✅ Risks identified con mitigations específicas
  • ✅ Un peer puede revisar y dar feedback constructivo basado en data, no opinions

Módulo 7 completado

Pasaste de tomar decisiones por intuición a tomarlas por framework. Concretamente:

  • ✅ Cinco trade-offs específicos (scale, cache, managed, serverless, routing)
  • ✅ Decision Matrix template reutilizable
  • ✅ Proyecto aplicado a sistema concreto

Lo que viene en M8 es integrar todo el path (M1-M7) en el Capstone Architecture Design final.

Siguiente módulo

Módulo 8 — Capstone Architecture Design. El proyecto integrador final. Tomás todo lo aprendido en M1-M7 y producís el blueprint completo del AI-Powered Knowledge Assistant. Es el documento más portfolio-worthy de la guía.

Recursos

  1. ADR examples by industry — ejemplos profesionales.
  2. Google SRE — Architecture decision documentation — patrones de documentación.
  3. Mozilla's architecture decision record template.
  4. DACI framework — para decisiones grupales.
  5. Stripe's tech radar — public examples of tech decisions.
← AnteriorDecision Matrix templateSiguiente →Introducción al Capstone Architecture Design