Módulo 8: Integrator Project — Capstone Architecture Design

Scaling + Reliability aplicado al Capstone

Descripción

M3 te enseñó scaling concepts. M5 reliability patterns. Esta cápsula los aplica al Capstone específico: ¿cuántas instancias de cada servicio? ¿Qué reliability patterns priorizar? ¿Cómo escalás de 5 tenants a 50?

Esta cápsula no introduce conceptos nuevos. Es la integración y specificity: decisiones concretas para el Knowledge Assistant.

Al terminar vas a poder:

Dimensionar capacity inicial y de scaling
Identificar bottlenecks específicos del Capstone
Priorizar reliability patterns (no implementar todo de una)
Diseñar capacity plan para 12 meses de growth

Sizing inicial

Para Year 1, 5 tenants iniciales con expectativa de 50:

Component	Initial sizing	Justification
Slack Adapter (FastAPI)	2 instancias Cloud Run	HA mínimo
Worker Pool	3 ECS instances, GPU-less initially	Pueden manejar ~10 req/min cada con LLM provider
Job Queue (Redis)	Single instance, 4GB	Suficiente para 5K jobs queued
MCP Gateway	2 instancias Cloud Run	HA
Each MCP server	1 instance Cloud Run	Bajos requisitos, scale por demanda
PostgreSQL	RDS db.t3.medium	Tenant data + thread history
Pinecone	Starter tier	~1M vectors total inicial
Cache Redis	Single instance, 2GB	Suficiente inicial

Estimated monthly cost (initial): ~$1,200-1,500/mes

Bottleneck analysis

Para identificar dónde primero te frenás:

Bottleneck candidates

LLM provider rate limits
- OpenAI Tier 4: 10,000 RPM
- At 100 req/min, no es bottleneck inmediato
- Becomes issue at 5,000+ RPM (50× current)
Worker capacity
- 3 workers × ~10 req/min = 30 req/min max
- At 30+ req/min, agregás workers
Pinecone queries
- Starter tier: limited QPS
- At growth phase, upgrade tier
Slack rate limits (outbound)
- 1 msg/seg per channel
- Bottleneck if mismo canal recibe muchas queries (rare en B2B)
MCP servers
- Cada uno con limits propios (GitHub API: 5000/hr per user, Notion: lower)
- Bottleneck si users hacen muchas queries usando mismo tool

Para sizing: identificá el bottleneck más cercano (worker pool en este caso) y diseñá scaling para el siguiente más cercano (Pinecone).

Auto-scaling configuration

Para cada servicio escalable:

Slack Adapter (Cloud Run)

# Cloud Run config
spec:
  template:
    spec:
      containerConcurrency: 80  # requests per instance
      timeoutSeconds: 30
    metadata:
      annotations:
        autoscaling.knative.dev/minScale: "2"
        autoscaling.knative.dev/maxScale: "20"
        autoscaling.knative.dev/targetCPUUtilizationPercentage: "70"

Trigger: CPU >70% o concurrent requests >80.

Worker Pool (ECS)

# ECS Service config
desiredCount: 3
minimumHealthyPercent: 50  # always at least 50% running during deploys
maximumPercent: 200

# Auto-scaling policy
targetTrackingScalingPolicy:
  targetValue: 70  # target CPU
  scaleInCooldown: 300  # 5 min before scale down
  scaleOutCooldown: 60  # 1 min before scale up

Trigger: CPU promedio >70%, o queue depth alerta.

Queue-based scaling (más important para AI)

Tradicional CPU scaling no detecta workers idle pero queue lleno. Custom metric:

# CloudWatch custom metric
def emit_queue_metric():
    queue_depth = redis.llen("knowledge-jobs")
    cloudwatch.put_metric_data(
        Namespace='KnowledgeAssistant',
        MetricData=[{
            'MetricName': 'QueueDepth',
            'Value': queue_depth,
        }]
    )

# Scaling policy basada en queue depth
{
    "MetricName": "QueueDepth",
    "TargetValue": 50,  # target: <50 messages in queue
    "ScaleOutCooldown": 60,
}

Cuando queue depth >50, agregar workers. Más responsive que CPU.

Reliability patterns: priorización

No vas a implementar todos los patterns de M5 en Day 1. Prioritizá:

Tier 1 (Day 1 — non-negotiable)

Pattern	Por qué
Queue-based processing	Sin esto, no podés cumplir 3s ACK de Slack
Signature verification (Slack)	Sin esto, abusable + no debería estar en internet
Idempotency (events)	Slack reentrega, sin esto procesás 3 veces
Multi-tenant filtering	Crítico para seguridad
Retry con backoff (LLM, MCP)	Errores transitorios constantes

Tier 2 (primer mes)

Pattern	Por qué
Rate limiting outbound (Slack)	Para no recibir 429
Circuit breaker (LLM provider)	Para detectar OpenAI outages
Health checks deep	Para auto-scaling correcto
Monitoring básico (Prometheus + Datadog)	Para detectar problemas

Tier 3 (Quarter 2)

Pattern	Por qué
Graceful degradation	Cuando crece volumen, importa
Fallback provider (Anthropic)	Si OpenAI tiene problemas, no caer
Semantic cache	Cost reduction
Memory snapshot (Modal)	Para reduce cold starts
Dead letter queue + manual replay	Para jobs perdidos

Razón de la priorización: cada tier es "10× más complejo de implementar bien". Empezá simple, agregá complexity solo cuando data lo justifique.

Capacity plan: 12 meses

Year 1 growth typical:

Month	Tenants	Daily queries	Avg req/min	Key changes
1	5	500	0.6	Launch with Tier 1 reliability
3	10	1,500	1.7	Implement Tier 2 patterns
6	20	5,000	5.5	Scale workers to 4-5, add LLM fallback
9	35	15,000	17	Add semantic cache, scale Pinecone
12	50	30,000	35	Tier 3 patterns implemented

Capacity decisions por milestone

Month 1-3 (Tier 1, low scale):

2 Cloud Run + 3 ECS workers
$1,200/mes total infra

Month 4-6 (Tier 2, moderate scale):

Scale workers to 5
Add Anthropic Claude fallback (cost +20% LLM)
Monitor depth increases — add Pinecone tier
$2,500/mes infra

Month 7-9 (Tier 3 start, growing):

5 workers + semantic cache
Cache hit rate target 30% (reduces LLM cost 30%)
$3,500/mes infra

Month 10-12 (Tier 3 full, near scale target):

6-8 workers with autoscaling
Cache hit rate 40%+
Full monitoring + alerting
$5,000-7,000/mes infra

Critical metrics to track

Métrica → Alerta thresholds → Actionable response:

Metric	Threshold	Response
Queue depth	>50 for 2 min	Auto-scale workers
Worker CPU	>85% sustained	Scale up
LLM latency P95	>10s	Investigate provider
LLM error rate	>5%	Activate fallback provider
Slack 429 rate	>10/hour	Adjust rate limiter
Vector DB latency P95	>2s	Scale Pinecone or investigate
MCP server down	Health check fails	Circuit breaker → degradation
Cache hit rate	<25%	Investigate (queries no son repetitivas?)
Cost projection	>budget × 1.1	Investigate (bug? abuse? expansion?)

Disaster recovery scenarios

Scenarios + your response:

Scenario 1: OpenAI 4-hour outage

Circuit breaker abre → fallback a Anthropic Claude
Quality degradation acceptable (Claude similar quality)
Monitor cost (Claude 2× pricing)
After outage: re-enable OpenAI, monitor stability

Scenario 2: Pinecone down

Worker fails con error en RAG retrieval
Graceful degradation: agent informa "RAG temporalmente no disponible, respuesta basada en conocimiento general del LLM"
Cache hit rate aumenta (responder con cached results helpful)
Si outage >30 min: notify ops team

Scenario 3: Database (PostgreSQL) outage

Critical: sin DB, no podés lookup tenant tokens
All queries fail
Plan: read replica para reads + master-slave failover. RDS Multi-AZ handles automatically.

Scenario 4: Spike of 10× normal traffic

Auto-scaling kicks in (puede tomar 2-3 minutos)
Queue depth crece
Workers escalan a maximum
Si still over capacity: rate limit más estricto, defer queries no-críticos
LLM provider rate limits podrían ser bottleneck

Scenario 5: Costo unexpectedly 3× higher

Investigate: ¿bug que retry infinito? ¿abuse de un tenant? ¿LLM expanded usage?
Tighten rate limits temporalmente
Investigate logs por tenant
Communicate transparente if budget-related

Trampas comunes

Trampa 1 — Implementar Tier 3 patterns Day 1. Sobre-engineering. Tu equipo es chico, no podés operar todo. Start simple.

Trampa 2 — Sizing para peak hypothético. "Vamos a tener 1M req/mes". No, vas a empezar con 1K req/mes y crecer. Size for current + 6 months, not Year 5.

Trampa 3 — Sin runbook. Pasa algo, equipo no sabe qué hacer. Runbook con scenarios + responses obligatorio.

Trampa 4 — Métricas sin alertas. Tenés dashboard, nadie lo mira. Alertas accionables (no "alert spam").

Trampa 5 — Auto-scaling sin floor. Scale to 0 instancias, primer request paga cold start. Min 1-2 instancias.

Trampa 6 — Single point of failure no identificado. PostgreSQL en single AZ. Pinecone si solo tenés starter tier. Identifícalos, decidí trade-off.

Ejercicio

Para tu Capstone:

¿Cuál es tu sizing inicial (Month 1)?
¿Cuál es tu bottleneck más cercano? Justificá con números.
¿Qué Tier 1 reliability patterns son non-negotiable para Day 1?
¿Cuándo agregás Tier 2? Define triggers.
Diseñá tu runbook para Scenario 1 (OpenAI outage 4 horas)

Ver solución

Sizing inicial:
- 2 Slack Adapter instances (HA)
- 3 worker instances
- 1 Redis instance
- 1 PostgreSQL RDS db.t3.medium (Multi-AZ for HA)
- Pinecone Starter
- 2 MCP Gateway instances
- 1 instance per MCP server
- Cost: ~$1,200/mes
Bottleneck más cercano: Worker pool (3 instancias × 10 req/min = 30 req/min max). Mes 1 esperamos 1.5 req/min, mes 6 ~5.5 req/min, mes 12 ~35 req/min. Hit bottleneck mes 11-12.
Tier 1 Day 1:
- Queue-based processing
- Signature verification (Slack, MCP)
- Idempotency (event_id check)
- Multi-tenant filtering en RAG + MCP
- Retry exponential backoff on LLM y MCP calls
Tier 2 triggers:
- Rate limiting outbound: Day 1 (Tier 1 actually; minimal)
- Circuit breaker: cuando hit primer LLM outage (probable mes 2-3)
- Health checks deep: cuando agregás load balancer / auto-scaling (mes 4)
- Monitoring: Day 1 (basic Datadog), iterar a mes 3

Runbook OpenAI outage 4hr:

## Scenario: OpenAI Outage (4 hours)

### Detection
- PagerDuty alert: LLM error rate >50% for 5 minutes
- Slack notification to #ops
- Datadog dashboard: OpenAI circuit state

### Response (first 10 min)
1. Verify outage at status.openai.com
2. Check circuit breaker state: should be OPEN, fallback active
3. Verify Anthropic Claude responding via fallback
4. Monitor cost: Claude is ~2x more expensive

### Response (during outage)
5. Communicate to clients via in-app banner: "Service degraded but operational"
6. Monitor quality: Claude similar but not identical responses
7. Check budget impact: estimate 4hr × 2x cost
8. If budget exceeded threshold: tighten rate limits temporarily

### Response (post-outage)
9. When OpenAI returns: monitor for 30 min stability before closing circuit
10. Close circuit manually if needed via admin endpoint
11. Post-mortem within 48h: what worked, what didn't
12. Review fallback usage patterns for improvements

### Escalation
- 4+ hours: notify CTO
- 8+ hours: emergency executive call

Resumen

Aprendiste:

✅ Initial sizing con costo estimado
✅ Bottleneck analysis priorizado
✅ Auto-scaling configurations específicas
✅ Reliability patterns en 3 tiers (Day 1, primer mes, Quarter 2)
✅ Capacity plan 12 meses con triggers de escalado
✅ Disaster recovery scenarios + responses
✅ Métricas y alertas accionables
✅ Trampas: over-engineering, sin runbook, single points of failure

Checkpoint: si podés justificar tu sizing con números, identificar el bottleneck más cercano, y tener un runbook básico, estás listo.

Siguiente cápsula

07 — Stack decisions + cost projection. Cerramos el design con Decision Matrices aplicadas a las decisiones del Capstone, y cost projection completa para 12 meses.

Recursos

Site Reliability Engineering — Google — comprehensive resource.
Release It! 2nd Edition — patterns de resilience.
AWS Well-Architected Framework — framework.
Cloud Run scaling.
ECS auto-scaling.

← AnteriorIntegration Layer Siguiente →Stack decisions + cost projection