Module 8: Your First AI System Design

6. Final Exercise: Design Your Own AI System

Description

Congratulations! You've reached the end of the guide. In this final exercise, you're going to design your own AI system by applying everything you've learned.

Format:

  1. You choose one of 3 use cases (or propose your own).
  2. You follow the design process (requirements → pattern → architecture → stack → trade-offs → costs).
  3. You produce deliverables (a diagram, a stack selection, a cost estimate).

This is NOT code. It's architectural design (preparation for implementation).

Use Case Options

Case 1: A Customer Support Chatbot

Context:

  • An ecommerce company (clothing).
  • Currently, support is via email (a 24-48h response time).
  • They want a 24/7 chatbot for FAQs (shipping, returns, sizes).

Requirements:

  • Answering frequent FAQs (80% of queries).
  • Escalating to a human agent (20% of complex queries).
  • Supporting Spanish and English.
  • Expected volume: 10K requests/day (~300K/month).
  • Budget: <$1K/month.

Case 2: A Code Assistant (Code Helper)

Context:

  • A software development startup.
  • Developers constantly ask about the internal codebase (how X works, where Y is).

Requirements:

  • Answering questions about the codebase (Python, 500 files).
  • Citing specific files/lines (e.g. "see auth.py line 45").
  • Updatable (the code changes daily).
  • Expected volume: 2K requests/day (~60K/month).
  • Budget: <$500/month.

Case 3: A Support Ticket Classifier

Context:

  • A SaaS company (a CRM).
  • They receive 1K tickets/day (billing, technical, account, feedback).
  • Currently, humans classify them manually (slow, expensive).

Requirements:

  • Classifying tickets automatically (billing, technical, account, feedback).
  • Accuracy >90%.
  • Latency <1s (routing them immediately).
  • Volume: 1K tickets/day (~30K/month).
  • Budget: <$200/month.

Case 4: Propose Your Own

If you have your own idea, use it. Make sure to define:

  • The context (the company, the problem).
  • The functional requirements (what it must do).
  • The non-functional requirements (accuracy, latency, volume, budget).

The Design Process (8 Steps)

Step 1: Understand the Requirements

Functional requirements:

  • What must the system do?

Non-functional requirements:

  • Accuracy: 90%, 95%?
  • Latency: <1s, <3s, <10s?
  • Volume: 1K, 10K, 100K requests/month?
  • Budget: $100, $500, $1K/month?

Step 2: Select a Pattern

Evaluate the 4 patterns:

  1. A simple chatbot: The LLM without external docs (a general FAQ).
  2. RAG: The LLM + external docs (Q&A over specific docs).
  3. Agents: The LLM + multiple tools (complex tasks).
  4. A classifier: The LLM categorizes text (tickets, sentiment).

Decision: Which pattern is appropriate for your use case?

Step 3: Design the Architecture

Draw a high-level diagram:

User → Frontend → Backend → [Components] → Response

Identify the components:

  • Frontend (React, Vue).
  • Backend (FastAPI, Express).
  • LLM API (GPT-4, Claude, Llama 3).
  • Vector DB (Pinecone, Chroma) — if RAG.
  • DB (PostgreSQL, MongoDB).
  • Cache (Redis) — optional.

Step 4: Select the Stack

For each component, evaluate the options:

ComponentOptionsDecisionJustification
FrontendReact, Vue??
BackendFastAPI, Express??
LLMGPT-4, GPT-3.5, Claude, Llama 3??
Vector DBPinecone, Chroma, Weaviate??
DBPostgreSQL, MongoDB??
CacheRedis, Memcached??

Step 5: The Complete Flow

Describe the data flow step by step:

Example (RAG):

  1. The user asks a question in the frontend.
  2. The frontend sends a POST to /query on the backend.
  3. The backend embeds the query (OpenAI embeddings).
  4. The backend searches the vector DB (the top 5 docs).
  5. The backend builds the prompt (system + docs + query).
  6. The backend calls the LLM (GPT-4).
  7. The LLM generates a response.
  8. The backend returns the response to the frontend.
  9. The frontend displays the response.

Step 6: Evaluate the Trade-Offs

For each decision, identify the trade-off:

DecisionAlternativeTrade-offWhy
GPT-4GPT-3.5Quality vs Cost?
PineconeChromaManaged vs Self-hosted?
CacheNo cacheLatency vs Personalization?

Step 7: Estimate the Costs

Calculate the monthly cost:

Expected volume: ___K requests/month.

Cost per request:

  • Embeddings: ___ tokens × $___ / 1,000 = $___
  • LLM: ___ input tokens × $___ / 1,000 + ___ output tokens × $___ / 1,000 = $___
  • Total/request: $___

Total monthly cost:

ComponentCost
LLM API (OpenAI, Anthropic)$__
Vector DB (Pinecone, etc.)$__
Cache (Redis)$__
Hosting (Vercel, Railway)$__
Total$___/month

Does it meet the budget? Yes / No

If not: How do you optimize? (multi-model, cache, GPT-3.5 vs GPT-4).

Step 8: Identify the Risks

3 main risks:

  1. Risk 1: ___ (e.g. hallucinations).

    • Mitigation: ___ (e.g. use GPT-4, cite sources).

  2. Risk 2: ___ (e.g. outdated docs).

    • Mitigation: ___ (e.g. CI/CD re-ingestion).

  3. Risk 3: ___ (e.g. API downtime).

    • Mitigation: ___ (e.g. a fallback to OpenRouter).

Deliverables

By the end, you should have:

  1. A high-level architecture diagram (components, the flow).
  2. A technology stack table (component, technology, justification).
  3. A trade-offs table (decision, alternative, trade-off, why).
  4. A cost estimate (a breakdown by component, the total/month).
  5. Identified risks (the risk, the mitigation).

An Example Deliverable (Case 1: A Support Chatbot)

1. Architecture Diagram

User → React (the chat UI) → FastAPI + LangChain → Redis (an FAQ cache) → Pinecone (a vector DB with the knowledge base) → GPT-3.5-turbo (FAQ) / GPT-4 (complex) → PostgreSQL (conversations) → React

2. Technology Stack

ComponentTechnologyJustification
FrontendReactA mature ecosystem, chat UIs available
BackendFastAPI + LangChainPython, easy integration with OpenAI
LLMMulti-model: GPT-3.5 (FAQ), GPT-4 (complex)It optimizes cost (80% GPT-3.5, 20% GPT-4)
Vector DBPineconeManaged, scalable (300K requests/month)
EmbeddingsOpenAI text-embedding-3-smallA quality/cost balance
CacheRedisFrequent FAQs (a 40% hit rate)
DBPostgreSQLStoring the conversations
DeploymentVercel (frontend), Railway (backend)Easy, scalable

3. Trade-Offs

DecisionAlternativeTrade-offWhy
Multi-model (GPT-3.5 + GPT-4)Pure GPT-4Complexity vs CostA 70% saving (80% of simple queries use GPT-3.5)
Pinecone ($70/month)Chroma (free)Managed vs Self-hostedThe team has no experience with self-hosting, Pinecone scales better
Cache (Redis)No cacheLatency vs Personalization40% of queries are repetitive FAQs → an instant response

4. Cost Estimate

Volume: 300K requests/month.

  • Cache hit rate: 40% (120K instant, free).
  • Requests with RAG: 180K/month.
  • 80% GPT-3.5 (144K): $0.0003/request → $43.
  • 20% GPT-4 (36K): $0.027/request → $972.
ComponentCost
OpenAI (GPT-3.5 + GPT-4)$1,015
Pinecone$70
Redis$15
Hosting$30
Total$1,130/month

Result: It exceeds the budget ($1K/month) by $130.

Optimization:

  • Increase the cache hit rate (40% → 60%) → it reduces requests to 120K → OpenAI $677.
  • New total: $792/month ✅ (under budget).

5. Risks

  1. Hallucinations: The LLM makes up information.

    • Mitigation: GPT-4 for complex queries, citing sources, a system prompt "Only use the knowledge base".

  2. An outdated knowledge base: Products/policies change.

    • Mitigation: A CI/CD pipeline for weekly re-ingestion (docs → embeddings → Pinecone).

  3. OpenAI API downtime: If the API goes down, the chatbot goes down.

    • Mitigation: A fallback to OpenRouter (it uses Claude if OpenAI fails).

Final Reflection

Self-Assessment Questions

  1. Why did you choose that pattern? (a simple chatbot, RAG, agents, a classifier)
  2. What trade-offs did you identify? (cost vs latency vs quality)
  3. How did you optimize to meet the budget? (multi-model, cache, GPT-3.5 vs GPT-4)
  4. What risks did you identify and how do you mitigate them? (hallucinations, API downtime, etc.)

Next Steps

1. Implement (the Bootcamp)

This guide gave you the conceptual fundamentals.

Next step: The AI Engineering Bootcamp (hands-on implementation).

  • Weeks 1-2: Python + LLM APIs.
  • Weeks 3-4: RAG (LangChain, Pinecone).
  • Weeks 5-6: Agents.
  • Weeks 7-12: Real projects.

2. A Portfolio

Build 3 projects:

  1. A simple chatbot (GPT-3.5 + FastAPI).
  2. A RAG system (Q&A over docs).
  3. A classifier (tickets, sentiment).

Publish on GitHub + a blog post → a portfolio for your job search.

3. The Job Search

With the fundamentals + a portfolio:

  • Apply to "junior AI Engineer" positions.
  • Highlight your projects (a chatbot, RAG, a classifier).
  • In interviews, use the knowledge from this guide (trade-offs, costs, patterns).

Summary of Module 8

You learned:

  1. The components of an AI system: Frontend, backend, LLM, vectors, DBs, cache, monitoring.
  2. Common patterns: A simple chatbot, RAG, agents, a classifier.
  3. Trade-offs: Cost vs latency vs quality (you can't maximize all 3).
  4. A case study: A complete design of a Q&A system (step by step).
  5. An exercise: You designed your own AI system.

Summary of the Complete Guide

You've completed all 8 modules:

  1. Module 1: What is AI? (definitions, history, types).
  2. Module 2: Machine Learning (supervised, unsupervised, reinforcement).
  3. Module 3: Neural Networks (layers, activations, CNNs, RNNs).
  4. Module 4: Transformers (attention, encoder/decoder, why they revolutionized AI).
  5. Module 5: LLMs (tokenization, embeddings, the context window, parameters, a comparison).
  6. Module 6: The API ecosystem (providers, pricing, local vs cloud, aggregators).
  7. Module 7: AI Engineering (the role, differences from ML/Data Science, skills, the day to day).
  8. Module 8: AI system design (components, patterns, trade-offs, a case study).

Now you understand:

  • ✅ What AI, ML, Deep Learning, LLMs are (conceptually).
  • ✅ How Transformers and LLMs work (without math, but with depth).
  • ✅ What AI Engineering is (and how it differs from ML/Data Science).
  • ✅ How to design AI systems (architecture, stack, trade-offs, costs).

Congratulations!

You've completed the "AI Fundamentals for Engineers" guide.

You're now prepared for:

  1. The AI Engineering Bootcamp (hands-on implementation).
  2. Building projects (a chatbot, RAG, a classifier).
  3. A job search (junior AI Engineer).

Next step: Apply what you've learned. Build your first AI system!

Thank you for completing this guide! If you have feedback, share it so we can improve future versions.

Version: 1.0 (February 2026)
Author: The NIEVA Content Team
Contact: [Your contact info if applicable]

← Previous5. Case Study: Designing a Q&A System over Technical Documentation