Module 7: RAG and Semantic Search
3. RAG Architecture: Components and Flows
Overview
Here you design RAG's conceptual architecture: the components (vector DB, embeddings, LLM), the data flows (indexing, query), and how it all fits together.
The complete RAG architecture
┌────────────────────────────────────────────────┐
│ RAG SYSTEM │
├────────────────────────────────────────────────┤
│ │
│ PHASE 1: INDEXING (Offline) │
│ ┌──────────────────────────────────┐ │
│ │ Docs → Chunks → Embeddings → DB │ │
│ └──────────────────────────────────┘ │
│ │
│ PHASE 2: RETRIEVAL (Online) │
│ ┌──────────────────────────────────┐ │
│ │ Query → Embedding → kNN → Chunks │ │
│ └──────────────────────────────────┘ │
│ │
│ PHASE 3: AUGMENTATION (Online) │
│ ┌──────────────────────────────────┐ │
│ │ Query + Chunks → Prompt │ │
│ └──────────────────────────────────┘ │
│ │
│ PHASE 4: GENERATION (Online) │
│ ┌──────────────────────────────────┐ │
│ │ Prompt → LLM → Response │ │
│ └──────────────────────────────────┘ │
│ │
└────────────────────────────────────────────────┘
RAG's components
1. The Vector Database
Function: Storing the document embeddings
Examples:
- Pinecone (managed)
- Weaviate (open-source)
- Qdrant (open-source)
- FAISS (a library)
Data stored:
- Embeddings (1536D vectors)
- Metadata (source, page, timestamp)
- The original text (the chunks)
2. The Embedding Model
Function: Converting text into vectors
Examples:
- OpenAI
text-embedding-3-small(1536D) - OpenAI
text-embedding-3-large(3072D) - Cohere
embed-v3(1024D) - Sentence-BERT (768D, local)
Use:
- Indexing: Converting the documents into embeddings
- Query: Converting the user's question into an embedding
3. The LLM (Large Language Model)
Function: Generating an answer grounded in the context
Examples:
- GPT-4 Turbo (128K context)
- GPT-4o (128K context)
- Claude 3 Opus (200K context)
- Llama 3 (8K context, local)
Input: Prompt = the query + the retrieved chunks
Output: The generated answer
4. The Orchestration Layer
Function: Coordinating the full flow (retrieval → augmentation → generation)
Frameworks:
- LangChain (Python/JS)
- LlamaIndex (Python)
- Haystack (Python)
- Custom (FastAPI + the OpenAI SDK)
The detailed flow: From query to answer
Phase 1: Indexing (once, offline)
1. Document ingestion
Input: 1000 PDFs
Output: The extracted text
2. Chunking
Input: Long text
Output: 10K chunks (500 tokens each)
3. Embedding
Input: 10K chunks
Output: 10K embeddings (1536D)
API: OpenAI text-embedding-3-small
4. Storage
Input: 10K embeddings + metadata + text
Output: An index in Pinecone
Cost: ~$0.10 (10K chunks × 500 tokens = 5M tokens; 5,000 × $0.00002/1K tokens)
Time: 5-10 minutes
Phase 2: Retrieval (on every query)
User Query: "How do I use Pinecone with OpenAI?"
1. Query embedding
Input: "How do I use Pinecone with OpenAI?"
Output: [0.23, -0.45, ..., -0.34]
API: OpenAI text-embedding-3-small
Latency: 50-200ms
2. kNN search
Input: The query embedding
Vector DB: Pinecone
Top-K: 5
Output: The 5 most relevant chunks
Latency: 10-50ms
An example of the returned chunks:
- Chunk 1: "A guide to integrating Pinecone with OpenAI..." (score: 0.92)
- Chunk 2: "Code example: OpenAI embeddings..." (score: 0.88)
- Chunk 3: "FAQ: How do I connect Pinecone?" (score: 0.85)
- Chunk 4: "Initial Pinecone setup..." (score: 0.82)
- Chunk 5: "Common troubleshooting..." (score: 0.80)
Phase 3: Augmentation (building the prompt)
The constructed prompt:
"""
You are an assistant who is an expert in Pinecone and OpenAI.
Context (the official documentation):
[Chunk 1]
A guide to integrating Pinecone with OpenAI:
1. Generate embeddings with the OpenAI API
2. Upsert the embeddings to Pinecone
3. Query with cosine similarity
...
[Chunk 2]
Code example: OpenAI embeddings
```python
embedding = openai.Embedding.create(
input="text",
model="text-embedding-3-small"
)
...
[Chunks 3-5] ...
The user's question: How do I use Pinecone with OpenAI?
Instructions:
- Answer based ONLY on the context provided
- If the context doesn't contain the answer, say "I don't have enough information"
- Cite the sources where relevant """
**Prompt size:** ~2500 tokens (5 chunks × 500 tokens)
---
### **Phase 4: Generation (the LLM produces the answer)**
Input: The complete prompt (2500 tokens) LLM: GPT-4 Turbo Output:
"To use Pinecone with OpenAI, follow these steps:
-
Generate embeddings with OpenAI: Use the
text-embedding-3-smallmodel to convert your text into 1536-dimensional vectors.embedding = openai.Embedding.create( input="your text here", model="text-embedding-3-small" ) -
Upsert to Pinecone: Store the embeddings in your Pinecone index:
index.upsert(vectors=[(id, embedding, metadata)]) -
Query with cosine similarity: Search for similar vectors using cosine similarity (the default in Pinecone).
For more detail, see the official integration guide."
Latency: 2-5 seconds (generation)
---
## The total latency of the flow
- Query embedding: 50-200ms
- kNN search: 10-50ms
- Prompt construction: <10ms
- LLM generation: 2-5 seconds
Total: 2.1-5.3 seconds
**Optimizations:**
- Cache frequent query embeddings
- Stream the LLM output (the user sees the answer as it's generated)
- A faster model (GPT-3.5 vs GPT-4)
---
## The complete flow diagram
User Query ↓ [Embed Query] (OpenAI) ↓ [kNN Search] (Pinecone) ↓ [Top-K Chunks] ↓ [Build Prompt] (Query + Chunks) ↓ [LLM Generate] (GPT-4) ↓ Response to User
---
## Summary
**Key points:**
- **Architecture:** Vector DB + Embeddings + LLM + Orchestration
- **The flow:** Indexing (offline) → Retrieval → Augmentation → Generation
- **Components:** Pinecone, OpenAI embeddings, GPT-4, LangChain
- **Latency:** 2-5 seconds typical (with LLM generation)
---
**Next capsule:** `04-the-rag-flow-step-by-step.md` — A detailed example with real data.