Module 8: Your First AI System Design

2. The Components of an AI System: Frontend, Backend, LLM, Vectors and DBs

Next →

Description

To design an AI system, you first need to understand what components exist and how they connect.

Typical components:

  1. Frontend: The UI (a chat interface, a search bar).
  2. Backend: The API, business logic, orchestration.
  3. LLM API: GPT-4, Claude, Llama 3 (text generation).
  4. Vector Database: Pinecone, Weaviate, Chroma (for RAG).
  5. A traditional database: PostgreSQL, MongoDB (conversations, user data).
  6. Cache: Redis (it reduces latency, cost).
  7. Monitoring: Logs, metrics (errors, latency, cost).

In this lesson you'll understand each component and how they connect.

Component 1: Frontend

What it is: The user interface (what the user sees).

Examples:

  • A chat interface (ChatGPT-like).
  • A search bar + results (Google-like with AI).
  • A form (the user enters text, receives an answer).

Common technologies:

  • React: An SPA (Single Page Application), reusable components.
  • Next.js: React with SSR (Server-Side Rendering), SEO.
  • Vue.js, Svelte: Alternatives to React.

Responsibilities:

  • Capturing the user's input.
  • Sending a request to the backend (POST /chat).
  • Displaying the backend's response.

Component 2: Backend

What it is: The server that handles business logic and calls the LLM APIs.

Common technologies:

  • FastAPI (Python): Async, easy integration with the OpenAI SDK, LangChain.
  • Express.js (Node.js): JavaScript, a broad ecosystem.
  • Flask (Python): Simpler than FastAPI (but with fewer features).

Responsibilities:

  1. Receiving requests from the frontend (POST /chat).
  2. Authentication: Verifying an API key or JWT.
  3. Calling the LLM API: OpenAI, Anthropic (generating a response).
  4. Orchestration: RAG (search docs → the LLM), agents (tool calling).
  5. Storing data: Saving the conversations in a DB.
  6. Returning the response to the frontend.

Component 3: LLM API

What it is: An external service that generates text (GPT-4, Claude, Llama 3).

Options:

  • The OpenAI API: GPT-4, GPT-3.5-turbo.
  • The Anthropic API: Claude 3 (Opus, Sonnet, Haiku).
  • The Google AI API: Gemini.
  • Self-hosted: Llama 3 (Ollama, vLLM).
  • An aggregator: OpenRouter (access to multiple models with one API).

Responsibilities:

  • Receiving a prompt (input).
  • Generating text (output).
  • Charging per token (input + output).

An example call:

import openai

response = openai.ChatCompletion.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "Hello"}]
)

print(response.choices[0].message.content)

Component 4: Vector Database

What it is: A database specialized in similarity search (for RAG).

Why it exists:

  • Traditional DBs (PostgreSQL) search by exact match (e.g. name = "John").
  • Vector DBs search by semantic similarity (e.g. "How do I reset my password?" ~ "I forgot my password").

How it works:

  1. Ingesting the docs: Documentation → chunks → embeddings (vectors) → stored in the vector DB.
  2. Query: The user asks → embed the query → search for similar vectors (cosine similarity) → retrieve the docs.
  3. The LLM: Generating a response using the docs as context.

Common technologies:

  • Pinecone: Cloud, managed, easy to use.
  • Weaviate: Open-source, self-hosted or cloud.
  • Chroma: Local, free, ideal for development.
  • Qdrant, Milvus: Open-source alternatives.

Component 5: A Traditional Database

What it is: A relational or NoSQL database for structured data.

Uses:

  • Storing conversations (user_id, message, timestamp).
  • Storing user data (email, plan, settings).
  • Logs (requests, errors).

Common technologies:

  • PostgreSQL: Relational, robust, SQL.
  • MongoDB: NoSQL, documents (JSON), flexible.
  • SQLite: Local, simple (for development).

Component 6: Cache

What it is: Temporary storage of results in order to reduce latency and cost.

Example:

  • The user asks: "What is Python?"
  • The backend calls OpenAI → gets a response → saves it in the cache (key: "What is Python?", value: the response).
  • Another user asks the same thing → the backend looks in the cache → returns the response (without calling OpenAI).

Common technologies:

  • Redis: In-memory, very fast.
  • Memcached: An alternative to Redis.

When to use it:

  • Frequently asked questions (FAQ).
  • Responses that don't change (e.g. "What is AI?").

Trade-off:

  • ✅ It reduces latency (an instant response from the cache).
  • ✅ It reduces cost (you don't pay tokens to OpenAI).
  • ❌ Static responses (not personalized).

Component 7: Monitoring and Logging

What it is: Tools for monitoring the system (errors, latency, cost).

Key metrics:

  • Latency: Response time (P50, P95, P99).
  • Error rate: The % of requests that fail.
  • Cost: Tokens consumed, cost per day/month.
  • Volume: Requests per day/month.

Common technologies:

  • Logs: Console logs, files (the backend saves logs of each request).
  • Metrics: Datadog, New Relic, Prometheus (dashboards).
  • AI-specific: LangSmith, Helicone (tracking LLM calls, tokens, cost).

Why it matters:

  • Detecting errors (e.g. the OpenAI API is down).
  • Optimizing cost (e.g. "80% of the cost goes to GPT-4 on simple questions").
  • Improving quality (e.g. "20% of the responses have hallucinations").

How the Components Connect

Example: A Simple Chatbot

Flow:

User (Frontend) → POST /chat → Backend → OpenAI API → Response → Backend → Frontend → User

In detail:

  1. The user writes "Hello" in the frontend (React).
  2. The frontend sends a POST to /chat (body: {"message": "Hello"}).
  3. The backend (FastAPI) receives the request.
  4. The backend calls the OpenAI API (GPT-3.5, prompt: "Hello").
  5. OpenAI returns a response ("Hello! How can I help you?").
  6. The backend saves the conversation in PostgreSQL.
  7. The backend returns the response to the frontend.
  8. The frontend displays the response.

Example: A RAG System

Flow:

User query → Backend → Embed the query → Vector DB (search for similar docs) → Retrieve docs → LLM (generate a response with the docs) → Backend → Frontend

In detail:

  1. The user asks "How do I reset my password?" in the frontend.
  2. The frontend sends a POST to /query (body: {"question": "..."}).
  3. The backend embeds the query (the OpenAI embeddings API).
  4. The backend searches the vector DB (Pinecone) for similar docs (cosine similarity).
  5. The vector DB returns the top 5 most relevant docs.
  6. The backend builds the prompt: 
    System: You are an assistant. Use these documents to answer.
Documents: [doc1, doc2, doc3, doc4, doc5]
User: How do I reset my password?
  7. The backend calls the LLM (GPT-4) with the prompt.
  8. The LLM generates a response based on the docs.
  9. The backend returns the response to the frontend.
  10. The frontend displays the response.

Component Trade-Offs

Cloud vs Self-Hosted

AspectCloud (Pinecone, OpenAI)Self-Hosted (Chroma, Llama 3)
Setup5 minutes1-3 days
Cost (low volume)Low ($10-100/month)High ($500-2K/month for a GPU)
Cost (high volume)High ($5K-50K/month)Medium ($500-2K/month fixed)
MaintenanceNoneHigh (updates, monitoring)
PrivacyLow (data goes to the provider)High (data stays in your infrastructure)

Why this matters for an AI Engineer

1. Architecture design

Without understanding the components:

  • "I use the OpenAI API" (too simple).

With an understanding of the components:

  • Frontend (React) → Backend (FastAPI) → the OpenAI API (GPT-3.5) → PostgreSQL (conversations) → Redis (an FAQ cache) → Datadog (monitoring).

2. Trade-offs

Each component has alternatives:

  • LLM: GPT-4 vs GPT-3.5 vs Llama 3.
  • Vector DB: Pinecone vs Chroma.
  • Backend: FastAPI vs Express.

You need to justify your choices.

Summary

The components of an AI system:

  1. Frontend: The UI (React, Next.js).
  2. Backend: The API, orchestration (FastAPI, Express).
  3. LLM API: Text generation (OpenAI, Claude, Llama 3).
  4. Vector DB: Semantic search for RAG (Pinecone, Weaviate, Chroma).
  5. A traditional DB: Structured data (PostgreSQL, MongoDB).
  6. Cache: Reducing latency/cost (Redis).
  7. Monitoring: Logs, metrics (LangSmith, Datadog).

Flow (a simple chatbot):

  • User → Frontend → Backend → OpenAI → Response → Frontend.

Flow (RAG):

  • User query → Backend → Embed → Vector DB → Retrieve docs → LLM → Response → Frontend.

Trade-offs:

  • Cloud (easy, scalable) vs Self-hosted (privacy, a fixed cost).

Next step: Lesson 03: Common Patterns — Chatbot, RAG, agents, classifier.

← Previous1. Module Introduction: From Concepts to ApplicationNext →3. Common Patterns: Chatbot, RAG, Agents and Classifier