7. Search System Case Studies

Overview

Here you analyze 3 real case studies: RAG (Retrieval-Augmented Generation), e-commerce search, and technical support. Each one with its architecture, design decisions, and metrics.

---

Case 1: RAG (Retrieval-Augmented Generation)

Context:

A Q&A system over technical documentation (10K pages of docs).

Requirements:

• The user asks a question → the system searches for relevant docs → an LLM generates the answer
• Latency < 2 seconds
• High precision (incorrect answers are critical)

---

Architecture:

User Query: "how to configure Pinecone"
    ↓
1. Query embedding (OpenAI ada-002)
    ↓
2. kNN search (Pinecone, top-100)
    ↓
3. Reranking (Cohere Rerank, top-5)
    ↓
4. Context injection (5 chunks into the LLM)
    ↓
5. LLM generation (GPT-4)
    ↓
Response: "To configure Pinecone..."

---

Design decisions:

1. Chunking:

• Strategy: Paragraph-based (it respects the structure)
• Size: 600 tokens (a balance of context/precision)
• Overlap: 100 tokens

2. Embeddings:

• Model: OpenAI text-embedding-3-small (1536D)
• Cost: ~$0.01 per full re-indexing

3. Vector DB:

• Pinecone (managed, HNSW)
• 1 index, 30K chunks

4. Reranking:

• The Cohere Rerank API
• Top-100 → Top-5 (precision is critical)

5. LLM:

• GPT-4 Turbo
• Context: The top-5 chunks (3000 tokens)

---

Metrics:

Precision@5: 0.85 (85% of the top-5 are relevant)
Latency: 1.2s on average
Cost per query: $0.003 (embedding + rerank + LLM)

---

Optimizations:

1. Caching frequent queries:
   "what is Pinecone" → a pre-generated answer → 0 latency

2. Streaming the LLM response:
   The user sees the answer as it's being generated → the perception of lower latency

3. Metadata filtering:
   If the user specifies a section (e.g. "in the setup guide") → filter the chunks by metadata

---

Case 2: E-commerce Search

Context:

A catalog of 500K products (clothing, electronics, home goods).

Requirements:

• Mixed queries: exact ("iPhone 15 Pro") and conceptual ("phone with a good camera")
• Latency < 300ms
• Personalization (the user's history)

---

Architecture:

User Query: "gaming laptop"
    ↓
1. Run in parallel:
   a. Keyword search (Elasticsearch, BM25)
   b. Semantic search (Weaviate, embeddings)
    ↓
2. Hybrid fusion (RRF, k=60)
    ↓
3. Personalization (boost viewed products)
    ↓
4. Filters (price, brand, rating)
    ↓
The top-20 products

---

Design decisions:

1. A dual index:

• Elasticsearch: Names, descriptions (keyword)
• Weaviate: Embeddings of the descriptions (semantic)

2. Embeddings:

• Model: Sentence-BERT (768D, local)
• Cost: $0 (after setup)

3. Fusion:

• RRF (k=60)
• An automatic balance (no need to tune α by hand)

4. Personalization:

final_score = base_score × (1 + history_boost)

history_boost:
- Product viewed: 1.2x
- Product in the cart: 1.5x
- Product purchased: 0.8x (they already have it)

---

Metrics:

Precision@10: 0.72
CTR (Click-Through Rate): 18%
Conversion rate: 4.2%
Latency: 180ms on average

---

Optimizations:

1. Pre-computed embeddings:
   The product embeddings are generated offline (not on every query)

2. Edge caching:
   Frequent queries ("iPhone 15") cached in the CDN

3. A/B testing:
   Trying different values of the RRF k (50, 60, 70)

---

Case 3: A Technical Support System

Context:

A knowledge base with 50K historical tickets + 5K help articles.

Requirements:

• Mixed queries: exact IDs ("Ticket #12345") and descriptions ("payment problem")
• Suggest similar tickets (for the support agents)
• Latency < 500ms

---

Architecture:

User Query: "payment problem using a card"
    ↓
1. Detect the pattern:
   Does it contain "Ticket #"? → Keyword exact match
   Otherwise → Semantic search
    ↓
2. Semantic search (Qdrant, HNSW)
    ↓
3. MMR (diversify the results)
    ↓
4. Group by category (payment, login, technical)
    ↓
The top-10 similar tickets

---

Design decisions:

1. ID detection:

Regex: /Ticket #[0-9]+/
If it matches → an exact search in the relational DB (PostgreSQL)
Otherwise → semantic search

2. Embeddings:

• Model: OpenAI text-embedding-3-small
• Field: The ticket's title + description

3. MMR:

• λ = 0.6 (60% relevance, 40% diversity)
• Avoid 10 nearly identical tickets

4. Grouping:

The top-10 tickets:
- Category "Payment": 4 tickets
- Category "Card": 3 tickets
- Category "General": 3 tickets

---

Metrics:

MRR: 0.68 (the first relevant result is at position 1-2)
Ticket resolution time: -25% (agents find solutions faster)
Latency: 320ms on average

---

Optimizations:

1. Auto-tagging:
   An LLM labels tickets with categories (payment, login, technical) → metadata for filters

2. A feedback loop:
   The agent marks a ticket as "useful" or "not useful" → retrain the embeddings

3. Preloading common issues:
   The top-20 frequent problems preloaded (latency: 0ms)

---

Comparing the cases

Aspect	RAG	E-commerce	Support
Dataset	10K docs	500K products	50K tickets
Queries	Conceptual	Mixed	Mixed
Latency	1.2s	180ms	320ms
Method	Semantic + rerank	Hybrid (RRF)	Semantic + MMR
Personalization	No	Yes (history)	Yes (feedback)

---

Summary

Key points:

• RAG: Semantic + reranking (precision is critical)
• E-commerce: Hybrid (keyword + semantic, personalization)
• Support: ID detection + semantic + MMR (diversity)
• Common optimizations: Caching, a feedback loop, metadata

---

Next capsule: 08-capstone-exercise-6.md — Designing an architecture for a case.