Module 8: Integrator Project — Capstone Architecture Design

Scaling + Reliability applied to the Capstone

Overview

M3 taught you scaling concepts. M5, reliability patterns. This lesson applies them to the specific Capstone: how many instances of each service? Which reliability patterns to prioritize? How do you scale from 5 tenants to 50?

This lesson doesn't introduce new concepts. It's the integration and specificity: concrete decisions for the Knowledge Assistant.

By the end you'll be able to:

Size initial and scaling capacity
Identify the Capstone's specific bottlenecks
Prioritize reliability patterns (not implement everything at once)
Design a capacity plan for 12 months of growth

Initial sizing

For Year 1, 5 initial tenants with an expectation of 50:

Component	Initial sizing	Justification
Slack Adapter (FastAPI)	2 Cloud Run instances	Minimum HA
Worker Pool	3 ECS instances, GPU-less initially	Each can handle ~10 req/min with the LLM provider
Job Queue (Redis)	Single instance, 4GB	Enough for 5K jobs queued
MCP Gateway	2 Cloud Run instances	HA
Each MCP server	1 instance Cloud Run	Low requirements, scale on demand
PostgreSQL	RDS db.t3.medium	Tenant data + thread history
Pinecone	Starter tier	~1M vectors total initially
Cache Redis	Single instance, 2GB	Enough initially

Estimated monthly cost (initial): ~$1,200-1,500/mo

Bottleneck analysis

To identify where you slow down first:

Bottleneck candidates

LLM provider rate limits
- OpenAI Tier 4: 10,000 RPM
- At 100 req/min, it's not an immediate bottleneck
- Becomes an issue at 5,000+ RPM (50× current)
Worker capacity
- 3 workers × ~10 req/min = 30 req/min max
- At 30+ req/min, you add workers
Pinecone queries
- Starter tier: limited QPS
- At the growth phase, upgrade tier
Slack rate limits (outbound)
- 1 msg/sec per channel
- A bottleneck if the same channel receives many queries (rare in B2B)
MCP servers
- Each one with its own limits (GitHub API: 5000/hr per user, Notion: lower)
- A bottleneck if users make many queries using the same tool

For sizing: identify the nearest bottleneck (the worker pool in this case) and design scaling for the next-nearest one (Pinecone).

Auto-scaling configuration

For each scalable service:

Slack Adapter (Cloud Run)

# Cloud Run config
spec:
  template:
    spec:
      containerConcurrency: 80  # requests per instance
      timeoutSeconds: 30
    metadata:
      annotations:
        autoscaling.knative.dev/minScale: "2"
        autoscaling.knative.dev/maxScale: "20"
        autoscaling.knative.dev/targetCPUUtilizationPercentage: "70"

Trigger: CPU >70% or concurrent requests >80.

Worker Pool (ECS)

# ECS Service config
desiredCount: 3
minimumHealthyPercent: 50  # always at least 50% running during deploys
maximumPercent: 200

# Auto-scaling policy
targetTrackingScalingPolicy:
  targetValue: 70  # target CPU
  scaleInCooldown: 300  # 5 min before scale down
  scaleOutCooldown: 60  # 1 min before scale up

Trigger: average CPU >70%, or queue depth alert.

Queue-based scaling (more important for AI)

Traditional CPU scaling doesn't detect idle workers but a full queue. Custom metric:

# CloudWatch custom metric
def emit_queue_metric():
    queue_depth = redis.llen("knowledge-jobs")
    cloudwatch.put_metric_data(
        Namespace='KnowledgeAssistant',
        MetricData=[{
            'MetricName': 'QueueDepth',
            'Value': queue_depth,
        }]
    )

# Scaling policy based on queue depth
{
    "MetricName": "QueueDepth",
    "TargetValue": 50,  # target: <50 messages in queue
    "ScaleOutCooldown": 60,
}

When queue depth >50, add workers. More responsive than CPU.

Reliability patterns: prioritization

You're not going to implement all of M5's patterns on Day 1. Prioritize:

Tier 1 (Day 1 — non-negotiable)

Pattern	Why
Queue-based processing	Without it, you can't meet Slack's 3s ACK
Signature verification (Slack)	Without it, abusable + it shouldn't be on the internet
Idempotency (events)	Slack redelivers, without it you process 3 times
Multi-tenant filtering	Critical for security
Retry with backoff (LLM, MCP)	Constant transient errors

Tier 2 (first month)

Pattern	Why
Outbound rate limiting (Slack)	To avoid getting 429
Circuit breaker (LLM provider)	To detect OpenAI outages
Deep health checks	For correct auto-scaling
Basic monitoring (Prometheus + Datadog)	To detect problems

Tier 3 (Quarter 2)

Pattern	Why
Graceful degradation	When volume grows, it matters
Fallback provider (Anthropic)	If OpenAI has problems, don't go down
Semantic cache	Cost reduction
Memory snapshot (Modal)	To reduce cold starts
Dead letter queue + manual replay	For lost jobs

Reason for the prioritization: each tier is "10× harder to implement well". Start simple, add complexity only when data justifies it.

Capacity plan: 12 months

Typical Year 1 growth:

Month	Tenants	Daily queries	Avg req/min	Key changes
1	5	500	0.6	Launch with Tier 1 reliability
3	10	1,500	1.7	Implement Tier 2 patterns
6	20	5,000	5.5	Scale workers to 4-5, add LLM fallback
9	35	15,000	17	Add semantic cache, scale Pinecone
12	50	30,000	35	Tier 3 patterns implemented

Capacity decisions per milestone

Month 1-3 (Tier 1, low scale):

2 Cloud Run + 3 ECS workers
$1,200/mo total infra

Month 4-6 (Tier 2, moderate scale):

Scale workers to 5
Add Anthropic Claude fallback (cost +20% LLM)
Monitor depth increases — add Pinecone tier
$2,500/mo infra

Month 7-9 (Tier 3 start, growing):

5 workers + semantic cache
Cache hit rate target 30% (reduces LLM cost 30%)
$3,500/mo infra

Month 10-12 (Tier 3 full, near scale target):

6-8 workers with autoscaling
Cache hit rate 40%+
Full monitoring + alerting
$5,000-7,000/mo infra

Critical metrics to track

Metric → Alert thresholds → Actionable response:

Metric	Threshold	Response
Queue depth	>50 for 2 min	Auto-scale workers
Worker CPU	>85% sustained	Scale up
LLM latency P95	>10s	Investigate provider
LLM error rate	>5%	Activate fallback provider
Slack 429 rate	>10/hour	Adjust rate limiter
Vector DB latency P95	>2s	Scale Pinecone or investigate
MCP server down	Health check fails	Circuit breaker → degradation
Cache hit rate	<25%	Investigate (queries aren't repetitive?)
Cost projection	>budget × 1.1	Investigate (bug? abuse? expansion?)

Disaster recovery scenarios

Scenarios + your response:

Scenario 1: OpenAI 4-hour outage

Circuit breaker opens → fallback to Anthropic Claude
Quality degradation acceptable (Claude similar quality)
Monitor cost (Claude 2× pricing)
After outage: re-enable OpenAI, monitor stability

Scenario 2: Pinecone down

Worker fails with an error in RAG retrieval
Graceful degradation: the agent informs "RAG temporarily unavailable, response based on the LLM's general knowledge"
Cache hit rate increases (answering with cached results helpful)
If outage >30 min: notify ops team

Scenario 3: Database (PostgreSQL) outage

Critical: without the DB, you can't look up tenant tokens
All queries fail
Plan: read replica for reads + master-slave failover. RDS Multi-AZ handles it automatically.

Scenario 4: Spike of 10× normal traffic

Auto-scaling kicks in (can take 2-3 minutes)
Queue depth grows
Workers scale to maximum
If still over capacity: stricter rate limit, defer non-critical queries
LLM provider rate limits could be a bottleneck

Scenario 5: Cost unexpectedly 3× higher

Investigate: a bug with infinite retry? abuse from one tenant? LLM expanded usage?
Tighten rate limits temporarily
Investigate logs per tenant
Communicate transparently if budget-related

Common traps

Trap 1 — Implementing Tier 3 patterns on Day 1. Over-engineering. Your team is small, you can't operate everything. Start simple.

Trap 2 — Sizing for a hypothetical peak. "We're going to have 1M req/mo". No, you're going to start with 1K req/mo and grow. Size for current + 6 months, not Year 5.

Trap 3 — No runbook. Something happens, the team doesn't know what to do. A runbook with scenarios + responses is mandatory.

Trap 4 — Metrics without alerts. You have a dashboard, nobody looks at it. Actionable alerts (not "alert spam").

Trap 5 — Auto-scaling without a floor. Scale to 0 instances, the first request pays a cold start. Min 1-2 instances.

Trap 6 — Unidentified single point of failure. PostgreSQL in a single AZ. Pinecone if you only have the starter tier. Identify them, decide the trade-off.

Exercise

For your Capstone:

What is your initial sizing (Month 1)?
What is your nearest bottleneck? Justify with numbers.
Which Tier 1 reliability patterns are non-negotiable for Day 1?
When do you add Tier 2? Define triggers.
Design your runbook for Scenario 1 (OpenAI outage 4 hours)

See solution

Initial sizing:
- 2 Slack Adapter instances (HA)
- 3 worker instances
- 1 Redis instance
- 1 PostgreSQL RDS db.t3.medium (Multi-AZ for HA)
- Pinecone Starter
- 2 MCP Gateway instances
- 1 instance per MCP server
- Cost: ~$1,200/mo
Nearest bottleneck: Worker pool (3 instances × 10 req/min = 30 req/min max). Month 1 we expect 0.6 req/min, month 6 ~5.5 req/min, month 12 ~35 req/min. Hit the bottleneck month 11-12.
Tier 1 Day 1:
- Queue-based processing
- Signature verification (Slack, MCP)
- Idempotency (event_id check)
- Multi-tenant filtering in RAG + MCP
- Retry exponential backoff on LLM and MCP calls
Tier 2 triggers:
- Outbound rate limiting: Day 1 (Tier 1 actually; minimal)
- Circuit breaker: when you hit the first LLM outage (probably month 2-3)
- Deep health checks: when you add a load balancer / auto-scaling (month 4)
- Monitoring: Day 1 (basic Datadog), iterate by month 3

Runbook OpenAI outage 4hr:

## Scenario: OpenAI Outage (4 hours)

### Detection
- PagerDuty alert: LLM error rate >50% for 5 minutes
- Slack notification to #ops
- Datadog dashboard: OpenAI circuit state

### Response (first 10 min)
1. Verify outage at status.openai.com
2. Check circuit breaker state: should be OPEN, fallback active
3. Verify Anthropic Claude responding via fallback
4. Monitor cost: Claude is ~2x more expensive

### Response (during outage)
5. Communicate to clients via in-app banner: "Service degraded but operational"
6. Monitor quality: Claude similar but not identical responses
7. Check budget impact: estimate 4hr × 2x cost
8. If budget exceeded threshold: tighten rate limits temporarily

### Response (post-outage)
9. When OpenAI returns: monitor for 30 min stability before closing circuit
10. Close circuit manually if needed via admin endpoint
11. Post-mortem within 48h: what worked, what didn't
12. Review fallback usage patterns for improvements

### Escalation
- 4+ hours: notify CTO
- 8+ hours: emergency executive call

Summary

You learned:

✅ Initial sizing with estimated cost
✅ Prioritized bottleneck analysis
✅ Specific auto-scaling configurations
✅ Reliability patterns in 3 tiers (Day 1, first month, Quarter 2)
✅ 12-month capacity plan with scaling triggers
✅ Disaster recovery scenarios + responses
✅ Actionable metrics and alerts
✅ Traps: over-engineering, no runbook, single points of failure

Checkpoint: if you can justify your sizing with numbers, identify the nearest bottleneck, and have a basic runbook, you're ready.

Next lesson

07 — Stack decisions + cost projection. We close the design with Decision Matrices applied to the Capstone's decisions, and a complete cost projection for 12 months.

Resources

Site Reliability Engineering — Google — comprehensive resource.
Release It! 2nd Edition — resilience patterns.
AWS Well-Architected Framework — framework.
Cloud Run scaling.
ECS auto-scaling.

← PreviousIntegration Layer Next →Stack decisions + cost projection