Module 1: System Design Thinking for AI

Introduction to System Design Thinking for AI

Next →

Capsule overview

You've completed 20 guides. You know how to build AI applications with FastAPI, containerize with Docker, deploy with CI/CD, monitor with metrics, optimize costs, and protect against prompt injection. You're able to take an empty repository and, in a few hours, have an AI system running in production. That ability — executing — is valuable. But there's another one you haven't practiced explicitly yet, and it's the one that separates a senior developer from an engineer who leads design: the ability to think before building.

This module opens the synthesis guide of the AI Engineering Path. Here you don't build new code. Here you learn to produce the artifacts that exist before the code: C4 diagrams that communicate architecture visually, Architecture Decision Records (ADRs) that document the why behind each choice, and structured trade-off analysis that makes design compromises explicit. These tools aren't bureaucracy — they're what separates a system designed intentionally from one that is the accidental sum of improvised decisions.

If you've ever inherited a system and spent weeks trying to understand why that database was chosen, why those services were split apart, why the queue is there — you've already suffered the absence of these artifacts. If you've ever had to defend an architectural decision in front of your CTO or a client and all you had was "because it seemed right at the time," same thing. This module gives you the tools so that your systems — and the systems you design as a team — don't cause that suffering to whoever comes next.

Where are we in the guide?

This is Guide #21: System Design & Scaling, part of the AI Engineering Path. It's the second-to-last guide of the path: only AI Ethics & Compliance (#22) remains before the Capstone. This position isn't accidental. You arrive here after learning every piece of the production stack (Docker, CI/CD, deployment, monitoring, cost optimization, security), and before implementing the path's integrative project (the AI-Powered Knowledge Assistant). The timing is perfect: you have the technical vocabulary to understand real trade-offs, and a concrete reason to apply what you learn here.

This module is the first of the three that make up Phase 1: Architectural Foundations. Here you establish the mindset and tools you'll use throughout the entire guide.

Phase 1: Architectural Foundations (Modules 1-3) ← YOU ARE HERE
  └── Module 1: System Design Thinking for AI (this module)
  └── Module 2: Common AI Architectures
  └── Module 3: Scaling Fundamentals

Phase 2: Integration & Patterns (Modules 4-6)
  └── Module 4: Integration Patterns (includes MCP)
  └── Module 5: Reliability at Scale
  └── Module 6: Real-World Integrations (Slack/Discord)

Phase 3: Trade-offs & Capstone Prep (Modules 7-8)
  └── Module 7: Performance vs Cost Trade-offs
  └── Module 8: Capstone Architecture Design

Each later module adds complexity to the design, but the mechanic of "analyze options → document decision → communicate with a diagram" that you establish here is the same in all of them. You're not learning M1 to forget it in M2 — you're installing the thought process that sustains the next 7 modules.

Why this guide is different

The first 20 guides taught you to do. This guide — and especially this module — teaches you to decide. The difference isn't trivial.

When you're doing, the questions are: how do I configure Redis? how do I write this endpoint? how do I deploy this container? The answers come from docs, tutorials, sample code. There's a correct solution (or at least several solutions that work), and your job is to find it and apply it.

When you're deciding, the questions are different: do I use a monolith or microservices? do I cache responses or recompute them? do I pay for a managed model or self-host? Here there's no correct answer — there are answers appropriate for your context and answers wrong for your context. What seems like the best option in the abstract can be the worst option for your team, budget, or constraints. This guide teaches you the method to discern which is which.

The correct answer to "which architecture do I use?" always starts with "it depends." But it's only a useful answer if it's followed by "it depends on these specific criteria, evaluated in this order, with these trade-offs accepted." Without those explicit criteria, "it depends" is an escape; with them, it's the beginning of a rigorous analysis. Learning those criteria is exactly what you're going to do.

Module objective

By the end of this module you'll be able to:

  • ✅ Articulate why AI systems require a different design approach than traditional web applications, citing the three fundamental constraints: latency variability, cost-per-request, and non-determinism
  • ✅ Perform structured trade-off analysis: given a requirement, identify at least two alternatives, list pros/cons of each, and justify the choice with explicit criteria
  • ✅ Create C4 diagrams at their three relevant levels: Context (system and external actors), Container (services and technologies), and Component (a service's internal modules)
  • ✅ Write a complete Architecture Decision Record (ADR): context, options evaluated, decision made, consequences accepted
  • ✅ Identify the specific constraints of a hypothetical AI system (latency budget, cost budget, determinism requirements) and map them to design decisions
  • ✅ Distinguish between reversible decisions (easy to change later) and irreversible ones (costly to revert), and apply the appropriate level of analysis to each
  • ✅ Produce a complete Architecture Decision Document for a hypothetical AI system that includes C4 diagrams, at least 3 ADRs, and documented trade-off analysis

Prerequisites

For this module you need:

  • ✅ To complete guides #1-#20 of the AI Engineering Path (or have equivalent experience)
  • ✅ To have built and deployed at least one complete AI system (not just prototypes)
  • ✅ Familiarity with Docker, CI/CD, and monitoring concepts (you'll use them in examples)
  • ✅ Willingness to move from "execution" to "design" — this module doesn't have much code

You don't need:

  • ❌ Prior experience in system design (the mindset is taught from scratch)
  • ❌ Knowledge of C4 or ADRs (they're introduced in this module)
  • ❌ To have led teams (the tools are the same for individuals and teams)

If you come from another path or from the industry without having completed the previous guides: the difference will show in the examples. The capsules assume you know what a vector DB is, what RAG is, what tiktoken does, how a Docker health check works. If those terms feel foreign to you, consider reviewing the relevant modules before continuing.

The mindset shift

The biggest obstacle you'll have in this guide isn't technical — it's psychological. After 20 guides writing code, it's going to be tempting to "skip" design and start implementing. That temptation feels productive: seeing code run gives more dopamine than seeing a static diagram in a document.

But the productivity of skipping design is illusory. Every architectural decision you don't document becomes implicit debt: someone (you in 6 months, or a colleague) is going to have to reconstruct your reasoning from the code, without access to the context you had when you wrote it. Every trade-off you don't analyze becomes a surprise: "we didn't think scaling would be this expensive" or "we didn't consider the model changing every 6 months." And every architecture you don't communicate becomes silos: each team member builds their own mental model, and when those models diverge, bugs appear at the edges.

Designing before building isn't perfectionism. It's the method that teams which build complex systems use to avoid collapsing under their own complexity. Amazon has the famous "working backwards" — they start with the product press release and design backward. Google publishes RFCs before every significant change. Anthropic, OpenAI, and Microsoft (the companies defining the AI engineering standard in 2026) have formal design review processes. It's not because they're slow — it's because that process lets them be fast at scale.

In this module you learn that process. Not to implement it religiously on every change to your system (that would be unnecessary overhead for trivial changes), but to have it available when the decision warrants it. Knowing when to apply rigorous design and when to improvise is part of the skill — and you learn that here too.

Module roadmap

CapsuleTopicWhat you'll learn
01Introduction (this capsule)Context of the module and the guide, design mindset
02Three constraints unique to AILatency variability, cost-per-request, non-determinism — with real numbers
03Structured trade-off analysisHow to evaluate options, explicit criteria, justified position
04C4 diagrams — Context and ContainerCommunicate architecture visually, the two most important levels
05C4 diagrams — ComponentInternal detail of services, when to go down to this level
06Architecture Decision Records (ADRs)Document the "why," template, real examples
07Reversible vs irreversible decisionsHow much analysis to apply to each decision, avoid over-engineering
08Project: Architecture Decision DocumentComplete document for a hypothetical AI system

Learning flow

First you understand what makes AI systems different (capsule 02) — the three fundamental constraints that affect every decision you'll make in the next 7 modules. Then you learn the method you'll apply repeatedly to make decisions: structured trade-off analysis (capsule 03). With that in hand, you learn to communicate your decisions: first visually with C4 diagrams (capsules 04-05), then in structured prose with ADRs (capsule 06). Capsule 07 teaches you to calibrate how much rigor to apply — not every decision deserves a formal ADR — and you close with a project that integrates everything (capsule 08).

The progression is deliberate: constraints → method → communication → calibration → integration. If you skip capsule 02, you won't understand why the trade-offs in capsule 03 are different from those of a traditional web system. If you skip capsule 06, you'll have diagrams but no documentation. If you skip capsule 07, you're going to apply unnecessary overhead to trivial decisions. Each one builds on the previous ones.

The module project

The deliverable for this module is a complete Architecture Decision Document for a hypothetical AI system: a "Q&A API over internal documents with RAG." The system is complex enough to generate real trade-offs (it's not a trivial chatbot), but not so complex that you get lost in implementation details.

The document you're going to produce includes:

  • C4 diagrams at Context, Container, and Component levels (at least one of each)
  • Three ADRs documenting key decisions: LLM provider choice, chunking strategy, deployment model
  • Explicit trade-off analysis for each ADR
  • Constraint mapping to decisions: latency budget, cost budget, determinism requirements
  • Reversible/irreversible classification of each decision

This document isn't just the M1 deliverable — it's the first piece of the path toward the Capstone Architecture Design in M8. The tools you practice here (C4, ADRs, trade-off analysis) are exactly the ones you'll use to design the complete architecture of the AI-Powered Knowledge Assistant. You're not writing a throwaway document — you're practicing the deliverables you'll produce as a tech lead or architect in your career.

If you've seen Architecture Decision Documents from companies like Google, Microsoft, or Amazon on their engineering blogs, they have exactly this structure. It's not an academic format invented for this guide — it's the industry standard. Learning it here prepares you to recognize it and produce it in your job.

How to make the most of this module

Three specific recommendations to get the most out of it:

1. Resist the urge to code. Your instinct is going to be, in every capsule, to open an editor and start implementing what's being discussed. Don't do it. The capsules have design exercises (not code) precisely because the muscle you're developing is different. If you have to code something to process it, do it after completing the design exercise, not instead of it.

2. Apply the concepts to a real system you know. The module's examples use the hypothetical Q&A system. But if you work on a real AI system (at your job or in a personal project), apply each concept to that system in parallel. What would the C4 diagrams of your current system look like? What decisions would you write as ADRs if you had them documented? This parallel exercise cements the learning far more than just reading.

3. Don't expect "aha moments" like in other modules. In technical guides, aha moments come when something finally compiles or a test passes. In design, aha moments are more subtle: you realize you were defending a decision without criteria, or that two options that seemed equal are fundamentally different when you analyze them rigorously. Those moments are less visceral but just as transformative. Trust the process even if it feels abstract at first.

The quality standard for your deliverable

When you finish this module, your Architecture Decision Document should be good enough that a senior colleague can:

  • Understand the architecture without you explaining it verbally. If they need to ask you "but why did you choose X?", the document failed. The answer should be in the corresponding ADR.

  • Disagree with your specific decision without disagreeing with your process. Your ADR should be so clear that someone who would reach a different conclusion can say "I understand your reasoning, I would have prioritized this other criterion" — not "I don't understand how you got to this."

  • Implement the system without asking you again. The C4 diagrams should be specific enough that a developer can start structuring the code without additional design.

This standard is ambitious, but not unreachable. The tools you're going to learn (C4, ADRs, trade-off analysis) are designed precisely to produce documents that meet these criteria. Your job in this module is to practice them until they become automatic.

What you'll see when you finish

Picture yourself three weeks from now, after completing this guide. You're in a meeting with your team discussing a new feature: "we want to add semantic search capabilities to our assistant." Before this guide, your instinct would have been: "ok, let's add a vector DB and a new endpoint." After this guide, your instinct will be different. You're going to ask: "what's the latency budget? how many queries do we expect per day? what happens if the vector DB has downtime? managed or self-hosted? who keeps the embeddings up to date?"

Those questions aren't senior-engineer rhetoric — they're the reflexes this guide installs. And they're exactly the questions that your CTO, your manager, or your client need someone to ask before committing resources to a feature. The day you become the person who asks those questions on your team, you'll remember that reflex was installed in M1 of this guide.

Summary and next step

In this capsule you established the context of the guide and this module:

  • System Design & Scaling is the synthesis guide of the AI Engineering Path: it connects all the technical skills you built in guides #1-#20 with a rigorous design framework
  • This module (M1) installs the mindset and tools: C4 diagrams, ADRs, structured trade-off analysis
  • The mindset shift is from "executing" to "deciding" — it's not perfectionism, it's the method that teams which build AI systems at scale use to avoid collapsing
  • The project is a complete Architecture Decision Document that practices the industry-standard format

Before moving on to capsule 02, you should be able to:

  • Articulate why this guide exists between #20 (Security) and the Capstone
  • Name the three deliverables you'll learn to produce (C4, ADRs, trade-off analysis)
  • Explain why "it depends" is the correct answer to "which architecture do I use?", but only if it's followed by specific criteria

If any of these raises doubt, reread the corresponding section before continuing. These concepts are the context on which everything else is built.

In capsule 02 — Three constraints unique to AI — you're going to go deeper into what makes an AI system fundamentally different from a traditional web system. You're not going to read generic assertions; you're going to see real numbers: how much GPT-4's latency varies between identical requests, how much an average request costs in production, how deterministic (or not) an LLM's behavior is. Those three constraints — latency variability, cost-per-request, non-determinism — are the foundation of every architectural decision you'll make in the next 7 modules. Without those numbers in your head, the trade-offs in capsule 03 feel abstract. With them, they feel urgent.

Resources

Next →Three Constraints Unique to AI Systems