2. The Overall Architecture of a Search System

Overview

Here you design the complete architecture of a vector search system: components, data flows, typical technologies. It's the blueprint that guides the whole design.

---

The main components

┌─────────────────────────────────────────────────┐
│              SEARCH SYSTEM                      │
├─────────────────────────────────────────────────┤
│                                                 │
│  1. INDEXING (Offline)                          │
│     └─ Documents → Chunks → Embeddings → DB     │
│                                                 │
│  2. QUERY PROCESSING (Online)                   │
│     └─ Query → Embedding → kNN → Results        │
│                                                 │
│  3. RANKING & FILTERING                         │
│     └─ Reranking, metadata filters, score fusion│
│                                                 │
│  4. STORAGE                                     │
│     └─ Vector DB (Pinecone, Weaviate)           │
│     └─ Metadata DB (PostgreSQL, etc)            │
│                                                 │
└─────────────────────────────────────────────────┘

---

The complete flow

Phase 1: Indexing (offline, once)

1. Document ingestion
   └─ PDF, TXT, HTML, Markdown

2. Chunking (splitting into parts)
   └─ By paragraph, by token, sliding window

3. Embedding (converting to vectors)
   └─ The OpenAI API, Sentence-BERT

4. Storage
   └─ Vector DB: embeddings + metadata

---

Phase 2: Query processing (online, on every search)

1. The user's query
   └─ "how to use RAG with Pinecone"

2. Embedding the query
   └─ The OpenAI API → [0.23, -0.45, ...]

3. kNN search
   └─ The vector DB returns the top-K candidates (k=100)

4. Reranking (optional)
   └─ A cross-encoder model reorders the top-100 → top-10

5. Return the results
   └─ The top-10 most relevant chunks

---

Architecture diagram

┌─────────────────┐
│    Documents    │  (PDF, TXT, HTML)
└────────┬────────┘
         │
         v
┌─────────────────┐
│    Chunking     │  (Split into 500 tokens)
└────────┬────────┘
         │
         v
┌─────────────────┐
│  Embedding API  │  (OpenAI, Cohere)
└────────┬────────┘
         │
         v
┌─────────────────┐
│    Vector DB    │  (Pinecone, Weaviate)
│   + Metadata    │
└────────┬────────┘
         │
         │ (Full storage)
         │
         ├──────────────────────────┐
         │                          │
         v                          v
┌─────────────────┐        ┌─────────────────┐
│   User Query    │        │      Query      │
│   "RAG setup"   │───────>│    Embedding    │
└─────────────────┘        └────────┬────────┘
                                    │
                                    v
                           ┌─────────────────┐
                           │   kNN Search    │
                           │   (HNSW/IVF)    │
                           └────────┬────────┘
                                    │
                                    v
                           ┌─────────────────┐
                           │    Reranking    │
                           │    (optional)   │
                           └────────┬────────┘
                                    │
                                    v
                           ┌─────────────────┐
                           │     Top-K       │
                           │     Results     │
                           └─────────────────┘

---

Typical technologies by component

Indexing:

• Chunking: LangChain, LlamaIndex (text splitters)
• Embeddings: The OpenAI API, Cohere, Sentence-BERT
• Orchestration: Python scripts, Airflow, Prefect

Vector DB:

• Managed: Pinecone, Weaviate Cloud
• Self-hosted: Weaviate, Qdrant, Milvus
• Library: FAISS (no native persistence)

Query processing:

• API: FastAPI, Flask, Express.js
• Reranking: The Cohere Rerank API, a cross-encoder (Sentence-BERT)

Metadata DB (optional):

• Relational: PostgreSQL, MySQL
• NoSQL: MongoDB, DynamoDB

---

Expected latency

Indexing:

1K documents × 500 tokens on average
→ Embedding: ~30 seconds (the OpenAI API, batched)
→ Indexing in Pinecone: ~5 seconds
Total: ~35-45 seconds

Query:

Embedding the query: 50-200ms (the OpenAI API)
kNN search (HNSW): 10-50ms (Pinecone)
Reranking (optional): 100-300ms (Cohere Rerank)
Total: 160-550ms

Optimization: Cache frequent embeddings (repeated queries).

---

Scalability

A small dataset (< 10K docs):

• Local FAISS (free)
• Embeddings generated once

A medium dataset (10K-1M docs):

• Self-hosted Weaviate or Pinecone Starter
• An HNSW index

A large dataset (> 1M docs):

• Pinecone Production or Weaviate Cloud
• Sharding (multiple indexes)
• A hybrid IVF+HNSW

---

Summary

Key points:

• Architecture: Indexing (offline) + query processing (online)
• Components: Chunking, embeddings, vector DB, kNN, reranking
• Technologies: OpenAI, Pinecone/Weaviate, FastAPI, Cohere
• Latency: 160-550ms typical (with reranking)

---

Next capsule: 03-the-indexing-pipeline.md — Chunking strategies, metadata.