Module 8: Your First AI System Design

1. Module Introduction: From Concepts to Application

Next →

Description

Welcome to Module 8: Your First AI System Design. This is the final module of the guide, where you integrate everything you've learned.

So far you've learned:

  • Module 1: What AI is (concepts, history, types).
  • Module 2: Machine Learning (supervised, unsupervised, reinforcement, training vs inference).
  • Module 3: Neural Networks (layers, activations, backpropagation, CNNs, RNNs).
  • Module 4: Transformers (attention, encoder/decoder, why they revolutionized AI).
  • Module 5: LLMs (tokenization, embeddings, the context window, parameters, a model comparison).
  • Module 6: The API ecosystem (providers, pricing, local vs cloud, aggregators).
  • Module 7: AI Engineering (the role, differences from ML/Data Science, skills, the day to day).

Now: You apply all of this by designing complete AI systems.

What "Designing an AI System" Means

System design: Planning the high-level architecture before coding.

It includes:

  1. Components: Frontend, backend, an LLM API, a vector DB, cache, monitoring.
  2. Data flow: User input → processing → the LLM → response → frontend.
  3. The technology stack: GPT-4 vs Claude, Pinecone vs Chroma, FastAPI vs Express.
  4. Trade-offs: Cost vs latency vs quality.
  5. Cost estimation: Pricing per token, expected volume.

It does NOT include complete code. It's a conceptual design (preparation for implementation).

Why Design Before Coding

Without Design: Common Problems

A real example:

  • A developer starts a chatbot with GPT-4.
  • It works well (an MVP).
  • It scales to 100K users/month.
  • Cost: $10K/month (prohibitive for a startup).
  • Problem: They didn't consider the cost from the start.

A late solution:

  • Migrate to GPT-3.5 (30× cheaper).
  • Rewrite the code (different prompts, adjustments).
  • A loss of 2 weeks.

With Design: Avoiding Problems

The correct flow:

  1. Design: Evaluate GPT-4 vs GPT-3.5 (trade-off: quality vs cost).
  2. Decide: GPT-3.5 for simple (80%), GPT-4 for complex (20%) → a 70% saving.
  3. Implement: The code accounts for a multi-model strategy from the start.
  4. Scale: The system is already optimized for cost.

What You'll Do in This Module

Lessons 02-04: Design Fundamentals

  • Lesson 02: The components of an AI system (frontend, backend, LLM, vectors, DBs).
  • Lesson 03: Common patterns (chatbot, RAG, agents, classifier).
  • Lesson 04: Design trade-offs (cost vs latency vs quality).

Lesson 05: A Case Study

A complete design of a Q&A system:

  • Requirements → components → stack → trade-offs → cost estimation.

Lesson 06: Final Exercise

Design your own AI system:

  • You choose the use case (a chatbot, RAG, a classifier).
  • You design the high-level architecture (a diagram, components, the flow).
  • You select the stack (LLM, vector DB, framework).
  • You estimate the costs.
  • You identify the trade-offs.

Connection with the Bootcamp

This guide is conceptual. The bootcamp is hands-on (code).

Flow:

  1. This guide (Modules 1-8): Conceptual fundamentals + architectural design.
  2. The bootcamp (Weeks 1-12): Implementation (code, deployment, production).

The benefit of designing first:

  • In the bootcamp, you understand why each decision is made (not just how to code it).

Design Format

There's no single format. But it typically includes:

1. An Architecture Diagram

A simple example (a chatbot):

User → Frontend (React) → Backend (FastAPI) → OpenAI API (GPT-3.5) → Response → Frontend

2. Components and Technologies

ComponentTechnologyJustification
FrontendReactA simple UI, a chat interface
BackendFastAPIPython, async, easy integration with OpenAI
LLMGPT-3.5-turboA quality/cost balance (simple tasks)
DatabasePostgreSQLStoring conversations
DeploymentVercel (frontend), Railway (backend)Easy, scalable

3. Data Flow

  1. The user writes a message in the frontend.
  2. The frontend sends a POST request to the backend (/chat).
  3. The backend calls the OpenAI API (GPT-3.5).
  4. The backend receives the response, stores it in the DB.
  5. The backend sends the response to the frontend.
  6. The frontend displays the response.

4. Trade-offs

DecisionAlternativeTrade-offWhy
GPT-3.5GPT-4Quality vs CostSimple tasks (FAQ), GPT-3.5 is enough, 30× cheaper
VercelAWSSimplicity vs ControlAn MVP, Vercel is faster to deploy

5. Cost Estimation

  • Expected volume: 10K requests/month.
  • Average per request: 500 input tokens, 300 output tokens.
  • Pricing (GPT-3.5): $0.0005/1K input, $0.0015/1K output.
  • Cost/month:
    • Input: 10K × 500 × $0.0005 / 1,000 = $2.50
    • Output: 10K × 300 × $0.0015 / 1,000 = $4.50
    • Total: $7/month (OpenAI) + $20/month (hosting) = $27/month

Common Mistakes

1. Overengineering

Mistake: Designing a system for 1M users when you have 0.

Solution: Design for an MVP (1K-10K users), then scale.

2. Not considering costs

Mistake: Using GPT-4 without calculating the cost at scale.

Solution: Always estimate costs (volume × pricing).

3. Not identifying trade-offs

Mistake: Choosing GPT-4 without justifying it (vs GPT-3.5).

Solution: For each decision, identify the alternative and the trade-off.

Why this matters for an AI Engineer

1. Technical interviews

A common question: "Design a support chatbot."

Expectation:

  • Identifying the components (frontend, backend, LLM, DB).
  • Evaluating the trade-offs (GPT-4 vs GPT-3.5).
  • Estimating the costs.

Without preparation: A superficial answer ("I use the OpenAI API").

With preparation: A complete design (components, trade-offs, costs).

2. Real work

Reality: At work, you need to propose an architecture before coding.

Stakeholders ask:

  • How much does it cost?
  • How long does it take to implement?
  • Does it scale?

You need answers based on a design.

Summary

This module integrates everything:

  • Modules 1-7 (concepts) → Module 8 (architectural design).

What you'll do:

  • Learn the components of AI systems.
  • Recognize common patterns (chatbot, RAG, agents).
  • Evaluate trade-offs (cost vs latency vs quality).
  • Design a complete system (a case study + an exercise).

Design format:

  1. An architecture diagram.
  2. Components and technologies.
  3. Data flow.
  4. Identified trade-offs.
  5. Cost estimation.

Why it matters:

  • Technical interviews (a common question: design a system).
  • Real work (proposing an architecture before coding).

Next step: Lesson 02: The Components of an AI System — Frontend, backend, LLM, vectors, DBs.

← Previous6. Exercise: Is AI Engineering for You? (An Honest Self-Assessment)Next →2. The Components of an AI System: Frontend, Backend, LLM, Vectors and DBs