Module 2: Common AI Architectures

Module 2 Project: Architecture Comparison Document

Next →

Project overview

This is the second deliverable of the guide. You're going to produce an Architecture Comparison Document that evaluates the three architectures (monolith, microservices, event-driven, and hybrid combinations) for a hypothetical AI system, applying the decision framework from capsule 07. This document is what a tech lead would present to their CTO when proposing an architecture for a new system — not an academic analysis, but an artifact with data, scoring, and a defensible recommendation.

Unlike the M01 project (which focused on how to document an architecture), this project focuses on which architecture to choose and why. The structure of the document is similar but the focus is on the rigorous comparison of options. You're going to use everything learned in M02: AI-specific criteria, patterns per architecture, AI-specific considerations, and the decision framework.

As in M01, this document is a portfolio piece and a precursor to the Capstone Architecture Design (M8). In M8 the architectural decision for the AI-Powered Knowledge Assistant will be real; this project is the rehearsal with a hypothetical case, but one complex enough to generate real trade-offs.

Module recap

Before the project, recap M02:

  • Capsule 01: Introduction and the bias to combat (cultural pro-microservices)
  • Capsule 02: AI monolith — when it wins, how to structure it well
  • Capsule 03: AI microservices — when they're justified, anti-patterns
  • Capsule 04: Event-driven — AI-specific patterns, queue-based LLM, batch processing
  • Capsule 05: Hybrid architectures — the real pattern, 3 common archetypes
  • Capsule 06: AI-specific considerations — cold start, context, LLM duplication, GPU, vector DB co-location
  • Capsule 07: Decision framework — 7 criteria, weights, structured scoring

The project integrates everything. It's the end-to-end application of the framework with M02/02-06 as criteria inputs.

The hypothetical system: specifications

Product context

You work at a healthcare tech scale-up (~150 employees) that's building a new product: MedQ — a B2B SaaS platform that gives hospitals an AI assistant to answer medical staff questions about clinical protocols, drug interactions, and internal policies. Your tech lead asks you to design the architecture.

Functional requirements

The system must:

  • Receive queries from medical staff via multiple interfaces: internal web app, mobile app (iOS/Android), Microsoft Teams Slack-like chat (hospital clients use Teams)
  • Respond with citations: each response must cite source protocols/policies with a link to the original document
  • Support multi-tenancy: each hospital is a separate tenant with its own documents, permissions, configuration
  • Handle diverse query types: protocol queries (RAG over PDFs/Word docs), drug interaction queries (calls to external APIs like UpToDate), admin policy queries (RAG over internal wiki)
  • Complete audit trail: regulatory compliance requires exhaustive logging of every query, response, tools used, and resulting medical decision (if applicable)
  • Support a feedback loop: medical staff can mark responses as correct/incorrect, which feeds the evals training
  • Real-time updates: when a hospital uploads new protocols, they must be available within 1 hour

Non-functional requirements

  • Latency: p99 ≤ 6 seconds (clinicians in emergencies need a fast response)
  • Availability: 99.95% uptime (healthcare regulation demands high reliability)
  • Target volume:
    • Year 1: 50 hospitals × 200 staff × 5 queries/day = 50k queries/day
    • Year 2: 200 hospitals × 200 staff × 5 queries/day = 200k queries/day
  • Ingestion latency: documents available in RAG within 1 hour of upload
  • Multi-region: US-East, US-West, EU (data residency requirements)
  • Compliance: HIPAA in the US, GDPR in the EU
  • Cost: AI budget $30k/month year 1, scalable to $100k/month year 2

Constraints

  • Team: 18 engineers organized into 4 sub-teams
    • AI Core team (5 engs): RAG, agent, LLM
    • Platform team (4 engs): infra, DB, auth, multi-tenancy
    • Integrations team (4 engs): web app, mobile, MS Teams
    • Compliance team (3 engs): audit, security, encryption
    • 2 free-floating senior engineers
  • Existing stack: Python (predominant), TypeScript for frontend
  • Cloud: AWS multi-region
  • Time: 9 months to production launch

Expected document structure

Your Architecture Comparison Document must have this structure:

# Architecture Comparison Document
# MedQ — AI Healthcare Q&A Platform

## 1. Executive Summary
[1-2 pages: context, main criteria, recommendation]

## 2. System Context (C4 Level 1)
[Context diagram with all actors and external systems]

## 3. Constraints & Decision Criteria
[Section that extracts constraints from the context and maps them to framework criteria]

## 4. Architecture Options Considered

### 4.1 Option A: Modular Monolith with Async Workers
[Container diagram, description, criteria evaluation]

### 4.2 Option B: Microservices Architecture
[Container diagram with separate services, evaluation]

### 4.3 Option C: Event-driven Architecture
[Diagram with event backbone, evaluation]

### 4.4 Option D: Hybrid Architecture (Recommended)
[Pragmatic hybrid diagram, evaluation]

## 5. Decision Framework Application
[Complete table with scoring, weights, conclusion]

## 6. Recommendation & Rationale
[Complete ADR in Nygard format]

## 7. AI-Specific Considerations Applied
[How each consideration from M02/06 is handled in the recommended architecture]

## 8. Implementation Roadmap
[Phases to build the architecture, dependencies, milestones]

## 9. Risks & Mitigations
[Top 5 identified risks with mitigation plans]

Partial example: system scoring

To help you calibrate depth, here is partial scoring of the MedQ system applying the framework:

Partial section 5: Decision Framework Application

## 5. Decision Framework Application

### Scoring of the MedQ system

| Criterion | Score | Justification |
|----------|-------|---------------|
| Team size | 3 | 18 engineers in 4 sub-teams |
| Latency | 3 | p99 6s, within the sync range |
| Scaling diff | 4 | Embedding worker (batch GPU-intensive) vs API (low CPU); Multi-tenancy requires per-tenant scaling |
| Stack uniformity | 2 | Python predominant, TypeScript for frontend (expected, not an architectural concern) |
| Maturity | 2 | Pre-launch, MVP phase, iterating boundaries |
| Isolation | 5 | HIPAA + GDPR + multi-tenancy require strong isolation between tenants and between tenant data |
| Async | 4 | Doc ingestion async (1h SLA), audit trails async, feedback loop async |

### Weights of the MedQ context

| Criterion | Weight | Reason |
|----------|------|-------|
| Team size | 4 | 18 engineers in sub-teams allows moderate microservices |
| Latency | 4 | Clinical UX non-negotiable |
| Scaling diff | 4 | Multi-tenant + GPU embedding difference is significant |
| Stack uniformity | 2 | Python uniform for backend |
| Maturity | 3 | MVP but with a clear plan |
| Isolation | 5 | HIPAA + GDPR non-negotiable |
| Async | 4 | Multiple natural async flows |

### Scoring conclusion

- **Pure monolith**: scores in team size (3), isolation (5), and scaling diff (4) are too high. Pure monolith is NOT viable.
- **Extensive microservices**: the scores point in that direction, but team size 3 suggests extreme microservices would be over-engineering for 18 engineers. 5-7 well-delimited microservices, not 20.
- **Event-driven backbone**: the async requirement (4) and isolation (5) partially justify it, but some key flows are synchronous.
- **Hybrid**: the appropriate combination — modular monolith for the core AI flow + 3-5 microservices for specific components + event-driven for naturally async flows + multi-region deployment for compliance.

**Decision: Hybrid — Modular Core + Targeted Microservices + Event-driven Backbone**

Partial section 6: ADR

## 6. Recommendation & Rationale

### ADR-MEDQ-001: Architecture Style Selection

## Status: Accepted

## Date: 2026-05-08

## Context

[Executive summary of the scoring + relevant constraints]

## Decision

Adopt a hybrid architecture with three combined styles:

**Core (Modular Monolith in Python)**:
- Query Service: orchestrating user queries
- RAG Pipeline: retrieval + reranking + prompt building
- Agent Service: tool use, multi-step reasoning
- LLM Client: wrapping Anthropic/OpenAI APIs
- Cache Layer: Redis for response cache + semantic cache

**Extracted microservices**:
- Embedding Service (Python, GPU): batch document processing, GPU-intensive
- Integration Service (TypeScript): MS Teams, web app, mobile API gateway
- Audit Service (Python): compliance logging with tamper-proof storage
- Tenant Management Service (Python): multi-tenancy admin, configuration
- (Possibly) Drug Interaction Service: external APIs with aggressive caching

**Event-driven flows**:
- Document ingestion queue (SQS)
- Query analytics events (Kafka)
- Feedback loop events (Kafka)
- Audit log events (with at-least-once delivery)

**Multi-region**:
- US-East (primary), US-West (read replica), EU (data residency)
- Per-region deployment of the monolith + critical services

## Alternatives Considered

[For each alternative: specific reason for rejection]

**Pure monolith**: scaling diff and compliance isolation requirements aren't met. A single instance doesn't allow per-region data residency.

**Pure microservices (10+ services)**: 18 engineers can't maintain 15+ services effectively. Operational overhead exceeds benefits.

**Pure event-driven**: critical UX latency (p99 6s) requires synchronous flows for query/response.

## Consequences

**Positive**:
- Compliance via service isolation (Audit, Tenant Management)
- Differentiated scaling (Embedding GPU vs others)
- Teams with moderate autonomy (Integration team, Compliance team)
- Async where necessary, synchronous where latency matters
- Multi-region viable

**Negative (trade-offs)**:
- Moderate operational complexity (5-7 services instead of 1-2)
- Distributed tracing mandatory (not optional)
- 4-6 additional weeks of setup vs a pure monolith
- Communication overhead between teams for cross-service changes

## Review trigger

Reevaluate the architecture if:
- Volume exceeds 500k queries/day (may justify more microservices)
- The team grows to 40+ engineers (more extensive microservices)
- Compliance regulations change significantly
- Quality/Latency targets change

Your job: complete the document

You have the system spec, partial scoring, and an ADR example. Your deliverable is to complete the document applying all of M02.

Specific deliverables

  1. Executive Summary complete (1-2 pages)
  2. System Context (C4 Level 1) with a diagram and prose
  3. Constraints & Decision Criteria complete (all constraints listed)
  4. 4 architecture options with Container diagrams and honest evaluation of each one:
    • Option A: Modular Monolith with Async Workers
    • Option B: Microservices Architecture
    • Option C: Event-driven Architecture
    • Option D: Hybrid (recommended)
  5. Decision Framework Application complete with scoring and weights
  6. Recommendation & Rationale (complete ADR)
  7. AI-Specific Considerations applied to the chosen architecture
  8. Implementation Roadmap with ordered phases
  9. Risks & Mitigations (top 5 risks)

Self-evaluation rubric

Your document passes if it meets all of the following:

Structure (Layer A — hygiene):

  • The 9 sections present and labeled
  • 4 C4 Container diagrams (one per option)
  • ADR follows the standard Nygard format
  • Valid Markdown, functional links

Pedagogical quality (Layer B):

  • Honest scoring: each criterion with a score 1-5 and a justification specific to the system
  • Contextual weights: differentiated weights (not all at 3-4), justified for the MedQ context
  • Each architecture evaluated on its merit: no straw-men of the rejected ones
  • Detailed hybrid: if you recommend hybrid (expected), specify what goes to each style and why
  • AI considerations applied: the 5 (cold start, context, LLM dup, GPU, vector DB) addressed explicitly
  • Honest trade-offs: minimum 3 negatives in the decision
  • Implementation roadmap: phases with dependencies, not a flat list

For an outsider:

  • A healthcare tech CTO can read and approve/disapprove with confidence
  • The implementation team can start working without more design
  • Decisions are defensible before a compliance audit

Estimated time

  • Executive Summary + Context: 30-45 min
  • Constraints & Criteria: 30 min
  • 4 architecture options + diagrams: 90-120 min
  • Decision Framework Application: 45 min
  • Complete ADR: 45 min
  • AI considerations + Roadmap + Risks: 60-90 min

Total: 5-7 hrs for a solid document. If it takes less, you probably skipped depth.

Anti-patterns to avoid

1. Straw-men of the architectures not chosen. "Microservices obviously doesn't work because they're complex" — without honest scoring, it's propaganda. Each architecture deserves rigorous evaluation before discarding it.

2. Decision framework in 5 minutes. Quick scoring without reflection is a valueless exercise. If all your scores are 3 or 4 without justification, the framework produced no information.

3. Hybrid without specificity. "Let's go hybrid" isn't a decision — it's a category. Your hybrid must have specific components assigned to specific styles.

4. AI considerations as an afterthought. The 5 considerations (M02/06) are part of the decision criteria, not a section at the end that repeats obvious points.

5. Implementation roadmap as a wishlist. Tasks listed at random without order. A real roadmap has dependencies, milestones, and a "done" criterion per phase.

6. Vague risks. "Risk: the architecture may fail" isn't a risk. Real risks: "If the Anthropic API has an outage of 1+ hour, the whole system goes down because we don't have a fallback configured. Mitigation: implement a fallback to OpenAI gpt-4o-mini with lower quality but graceful degradation."

Summary and next step

This project integrates all of M02:

  • The decision framework is applied with real data from the MedQ system
  • 4 architectures evaluated rigorously, not discarded a priori
  • Hybrid recommendation with specificity of what goes to each style
  • AI considerations applied explicitly
  • Complete ADR that a CTO can approve with confidence

The document is:

  • Your second portfolio deliverable
  • The basis for similar deliverables in M3-M7
  • The precursor to the Capstone Architecture Design (M8)

Before moving on to Module 3, you should have:

  • A complete Architecture Comparison Document (~10-15 pages)
  • 4 architectures evaluated with the scoring framework
  • A standard-format ADR with honest trade-offs
  • An actionable implementation roadmap
  • Top 5 risks with mitigations

If your document meets the rubric, you completed M02 successfully. If something fails, go back to the corresponding capsule.

In Module 3 — Scaling Fundamentals — you're going to take the recommended architecture and ask yourself: "how do we scale it?" The connection is direct — the scaling strategy depends on the chosen architecture. A monolith scales differently than microservices. Event-driven has different scaling considerations (queue depth, consumer count). You're going to learn horizontal vs vertical scaling, identification of AI-specific bottlenecks (LLM inference, embedding computation, vector search), stateless design for AI, and appropriate auto-scaling triggers (queue depth, request rate) instead of just CPU/memory. Phase 1 (Architectural Foundations) closes when you combine M01 (how to think) + M02 (which architecture) + M03 (how to scale it) into a complete mental model.

Resources

← PreviousDecision Framework for Choosing an ArchitectureNext →Introduction to Scaling Fundamentals