5. Case Study: Designing a Q&A System over Technical Documentation

Description

In this lesson you'll see a complete design of an AI system, step by step:

The system: Q&A over the technical documentation of a SaaS product.

Process:

1. Understand the requirements.
2. Select a pattern (RAG).
3. Design the architecture.
4. Select the stack.
5. Evaluate the trade-offs.
6. Estimate the costs.

---

Step 1: Understand the Requirements

Context

Company: A SaaS startup (100 employees).

Problem: Developers constantly ask about the API docs → time lost searching.

Proposed solution: A chatbot that answers questions about the docs.

---

Functional Requirements

1. Answering questions about the docs: "How do I authenticate API requests?"
2. Supporting Spanish and English.
3. Citing sources: Showing which doc was used to answer.
4. Updatable: The docs change every week.

---

Non-Functional Requirements

1. Accuracy: >90% (correct answers).
2. Latency: <3s (tolerable for developers).
3. Volume: 1K requests/day (~30K/month).
4. Budget: <$500/month.

---

Step 2: Select a Pattern

Evaluation

Options:

1. A simple chatbot: The LLM without external docs.

   • ❌ It doesn't work (the LLM doesn't know the product's specific docs).

2. RAG: The LLM + external docs.

   • ✅ Perfect (docs → embeddings → a vector DB → the LLM).

3. Fine-tuning: Fine-tuning Llama 3 with the docs.

   • ❌ Complex, expensive, overkill (RAG is enough).

4. Agents: The LLM + tools (search, a DB).

   • ❌ Overkill (it doesn't need multiple tools).

Decision: RAG (the appropriate pattern for Q&A over docs).

---

Step 3: Design the Architecture

High-Level Diagram

User (Frontend) → POST /query → Backend (FastAPI + LangChain) 
  → Embed the query (OpenAI embeddings) 
  → Search the Vector DB (Pinecone) 
  → Retrieve the top 5 docs 
  → LLM (GPT-4) 
  → Response (with sources) 
  → Frontend

---

Components

Component	Function
Frontend	A chat interface (React)
Backend	RAG orchestration (FastAPI + LangChain)
The embeddings API	Converting the query/docs into vectors (OpenAI text-embedding-3-small)
Vector DB	Storing the docs' embeddings (Pinecone)
LLM API	Generating a response based on the docs (GPT-4)
DB	Storing the conversations (PostgreSQL)
Cache	Reducing latency/cost (Redis)

---

Step 4: Select the Stack

Evaluation by Component

4.1. Frontend

Options: React, Vue, Svelte.

Decision: React (a mature ecosystem, the team already knows it).

---

4.2. Backend

Options: FastAPI (Python), Express (Node.js).

Decision: FastAPI (Python, easy integration with the OpenAI SDK, LangChain).

---

4.3. Embeddings

Options:

• OpenAI text-embedding-3-small ($0.00002/1K tokens).
• OpenAI text-embedding-3-large ($0.00013/1K tokens, better quality).
• Sentence-Transformers (free, self-hosted).

Decision: OpenAI text-embedding-3-small (a quality/cost balance, 6× cheaper than large).

---

4.4. Vector DB

Options:

Vector DB	Pros	Cons
Pinecone	Cloud, managed, scalable	Expensive ($70/month starter)
Weaviate	Open-source, cloud or self-hosted	A more complex setup
Chroma	Free, local	It doesn't scale well (>100K docs)

Decision: Pinecone (managed, scalable, the team has no experience with self-hosting).

Trade-off: More expensive ($70/month), but less maintenance.

---

4.5. LLM

Options:

LLM	Accuracy	Cost/1K tokens (input)	Latency
GPT-4	95%	$0.03	500ms
GPT-3.5-turbo	85%	$0.0005 (60× cheaper)	300ms
Claude 3 Opus	95%	$0.015 (2× cheaper than GPT-4)	400ms

Initial decision: GPT-4 (accuracy is critical).

Later optimization: Multi-model (GPT-3.5 for a simple FAQ, GPT-4 for complex ones).

---

4.6. Cache

Options: Redis, Memcached.

Decision: Redis (in-memory, very fast, a configurable TTL).

Strategy:

• Cache frequent queries (FAQ).
• TTL: 7 days (the docs are updated weekly).

---

Step 5: The Complete Flow

5.1. Ingestion (Once a Week)

Process:

1. Docs (Markdown) → Parse → Chunks (512 tokens each).
2. Chunks → the OpenAI embeddings API → Vectors.
3. Vectors → Pinecone (stored with metadata: doc_id, source_url).

Example:

# Pseudo-code
docs = load_docs("docs/")  # Load Markdown files
chunks = split_docs(docs, chunk_size=512)  # Split into chunks

for chunk in chunks:
    embedding = openai.Embedding.create(input=chunk.text, model="text-embedding-3-small")
    pinecone.upsert(id=chunk.id, vector=embedding, metadata=chunk.metadata)

---

5.2. Query (Each Request)

Process:

1. User: "How do I authenticate API requests?"
2. Frontend → POST /query → Backend.
3. The backend checks the cache (Redis):
   • Cache hit: It returns the response (instant, free).
   • Cache miss: It continues with RAG.
4. The backend embeds the query (OpenAI embeddings).
5. The backend searches Pinecone (the top 5 docs by cosine similarity).
6. The backend builds the prompt:

   System: You are a technical assistant. Use these documents to answer. Cite your sources.
   Documents:
   1. [doc_1: "API Authentication requires Bearer token..."]
   2. [doc_2: "Generate token at /api/auth..."]
   ...
   User: How do I authenticate API requests?
   

7. The backend calls GPT-4.
8. GPT-4 answers:

   To authenticate, use a Bearer token in the Authorization header. 
   Generate the token at /api/auth. (Source: doc_1, doc_2)
   

9. The backend stores it in the cache (Redis, TTL: 7 days).
10. The backend returns the response to the frontend.
11. The frontend displays the response (with the sources).

---

Step 6: Evaluate the Trade-Offs

Trade-Off 1: GPT-4 vs GPT-3.5

Aspect	GPT-4	GPT-3.5
Accuracy	95%	85%
Cost/request	$0.006	$0.0001 (60× cheaper)
Cost/month (30K)	$180	$3

Initial decision: GPT-4 (accuracy is critical for technical docs).

Future optimization: Multi-model (a classifier detects complexity → GPT-3.5 for simple, GPT-4 for complex).

---

Trade-Off 2: Pinecone vs Chroma

Aspect	Pinecone	Chroma
Cost	$70/month	Free
Maintenance	None (managed)	High (self-hosted)
Scalability	High (millions of docs)	Medium (100K docs max)

Decision: Pinecone (the team has no experience with self-hosting, future scalability).

Accepted trade-off: $70/month more expensive, but less maintenance.

---

Trade-Off 3: Latency vs Accuracy

Aspect	With Cache	Without Cache
Latency (cache hit)	<50ms	N/A
Latency (cache miss)	N/A	2-3s
Accuracy	Static (a pre-generated response)	Dynamic (personalized)

Decision: Cache for frequent queries (FAQ) → 40% of requests hit the cache → latency <50ms.

Accepted trade-off: Static responses on the FAQ (but users tolerate it, since these are frequent questions).

---

Step 7: Estimate the Costs

Expected Volume

• 30K requests/month.
• Cache hit rate: 40% (12K requests instant, free).
• Requests with RAG: 18K/month.

---

Cost per Request (RAG)

Embeddings (the query):

• Query: ~50 tokens.
• Cost: 50 × $0.00002 / 1,000 = $0.000001 (negligible).

Vector DB (Pinecone):

• $70/month (a flat fee, not per request).

LLM (GPT-4):

• Input: 500 tokens (system + docs + query).
• Output: 200 tokens (the response).
• Cost: (500 × $0.03 + 200 × $0.06) / 1,000 = $0.027/request.

---

Total Monthly Cost

Component	Cost
OpenAI (GPT-4)	18K × $0.027 = $486
Pinecone	$70
Redis	$15
Hosting (Vercel + Railway)	$30
Total	$601/month

Result: It exceeds the budget ($500/month) by $101.

---

Optimization to Meet the Budget

Option 1: Use GPT-3.5 for simple queries

• A classifier detects complexity (GPT-3.5, $0.0001/request).
• 60% simple queries → GPT-3.5 ($0.0003/request).
• 40% complex queries → GPT-4 ($0.027/request).
• New OpenAI cost: (18K × 0.6 × $0.0003) + (18K × 0.4 × $0.027) = $198.
• Total: $198 + $70 + $15 + $30 = $313/month ✅ (under budget).

---

Step 8: Identify the Risks

Risk 1: Hallucinations

Problem: The LLM makes up information.

Mitigation:

• Use GPT-4 (fewer hallucinations than GPT-3.5).
• A system prompt: "Only use information from the documents. If you don't know, say 'I don't have information about that'".
• Cite sources (users verify).

---

Risk 2: Outdated Docs

Problem: The docs are updated every week, the embeddings aren't.

Mitigation:

• A CI/CD pipeline: Every doc update → automatic re-ingestion (embeddings → Pinecone).

---

Risk 3: OpenAI API Downtime

Problem: If the API goes down, the system goes down.

Mitigation:

• Fallback: OpenRouter (it uses Anthropic Claude if OpenAI goes down).

---

Design Summary

Final Architecture

User → React → FastAPI + LangChain → Redis (cache) → OpenAI embeddings → Pinecone → GPT-4 (or GPT-3.5) → PostgreSQL → React

---

Final Stack

Component	Technology	Justification
Frontend	React	A mature ecosystem
Backend	FastAPI + LangChain	Python, easy integration
Embeddings	OpenAI text-embedding-3-small	A quality/cost balance
Vector DB	Pinecone	Managed, scalable
LLM	Multi-model (GPT-3.5 + GPT-4)	It optimizes cost (60% GPT-3.5, 40% GPT-4)
Cache	Redis	Reducing latency/cost (a 40% hit rate)
DB	PostgreSQL	Storing the conversations
Deployment	Vercel (frontend), Railway (backend)	Easy, scalable

---

Final Costs

• Total: $313/month (under the $500/month budget).
• Breakdown: OpenAI ($198) + Pinecone ($70) + Redis ($15) + Hosting ($30).

---

Accepted Trade-Offs

1. Pinecone (expensive) vs Chroma (free): Paying $70/month for a managed service.
2. Multi-model (complex) vs pure GPT-4 (simple): Additional complexity (a classifier) → a 60% saving.
3. Cache (static responses) vs no cache (personalized): 40% of requests are static (FAQ) → savings in latency/cost.

---

Why this matters for an AI Engineer

1. A systematic process

This case study shows how to design step by step:

1. Requirements → Pattern → Architecture → Stack → Trade-offs → Costs.

---

2. Communicating with stakeholders

The Product Manager asks: "How much does it cost?"

The AI Engineer (with a design): "$313/month (OpenAI $198, Pinecone $70, others $45)."

---

Summary

A complete design of a Q&A system over docs:

1. Requirements: Accuracy >90%, latency <3s, 30K requests/month, a $500/month budget.
2. Pattern: RAG (docs → embeddings → a vector DB → the LLM).
3. Architecture: React → FastAPI → Redis → OpenAI embeddings → Pinecone → GPT-4/GPT-3.5 → PostgreSQL.
4. Trade-offs: Pinecone (managed) vs Chroma (free), multi-model (complex) vs pure GPT-4, cache (static) vs no cache.
5. Costs: $313/month (under budget).
6. Risks: Hallucinations (mitigation: GPT-4, citing sources), outdated docs (CI/CD re-ingestion), API downtime (an OpenRouter fallback).

Next step: Lesson 06: Final Exercise — Design your own AI system.