3. The Indexing Pipeline: Chunking and Embeddings

Overview

Indexing is the offline process of turning raw documents into stored vectors. This capsule covers chunking strategies (how to split documents), embedding generation, and storage with metadata.

---

Step 1: Chunking (splitting documents)

Why chunking?

• Long documents (10K+ tokens) don't fit in an embedding
• Small chunks → more precision in search
• Large chunks → more context

---

Strategy 1: Fixed-size chunks

Document: 5000 tokens
Chunk size: 500 tokens
Overlap: 50 tokens

Result:
- Chunk 1: tokens 0-500
- Chunk 2: tokens 450-950 (a 50-token overlap)
- Chunk 3: tokens 900-1400
...

Advantage: Simple, predictable.
Disadvantage: It can cut sentences in half.

---

Strategy 2: Paragraph-based

Split on:
- A double line break (\n\n)
- A token limit per chunk (e.g. a maximum of 800)

Result:
- Chunk 1: Paragraphs 1-3 (500 tokens)
- Chunk 2: Paragraphs 4-5 (400 tokens)

Advantage: It respects the semantic structure.
Disadvantage: Chunks of variable size.

---

Strategy 3: Sentence-based

Split on sentences (. ! ?)
Group them up to a maximum of N tokens

Advantage: It doesn't cut sentences.
Disadvantage: Very small chunks (less context).

---

Strategy 4: Hierarchical (recursive)

1. Split by section (# headers in Markdown)
2. If a section is > 800 tokens → split it by paragraph
3. If a paragraph is > 800 tokens → split it by sentence

Advantage: It respects the hierarchical structure.
Disadvantage: More complex to implement.

---

Recommended chunk size

Use	Chunk size	Justification
General RAG	500-800 tokens	A balance of context/precision
Code search	200-400 tokens	Complete functions
Technical documentation	800-1200 tokens	Complete sections
Long articles	1000-1500 tokens	Maintaining the narrative

Rule of thumb: 500-800 tokens with an overlap of 50-100.

---

Overlap

Why overlap?

Without overlap:

Chunk 1: "The dog is loyal."
Chunk 2: "Cats are independent."

Query: "the difference between dogs and cats"
→ No chunk contains both ❌

With overlap:

Chunk 1: "The dog is loyal. Cats..."
Chunk 2: "...dog is loyal. Cats are independent."

Query: "the difference between dogs and cats"
→ Chunk 2 contains the full context ✅

Recommended overlap: 10-20% of the chunk size (50-100 tokens for 500-token chunks).

---

Step 2: Metadata extraction

Useful metadata:

• source: The name of the original document
• page: The page number (for PDFs)
• section: The section/chapter
• timestamp: The creation/update date
• author: The document's author
• tags: Manual or automatic labels

Example:

{
  "id": "chunk-123",
  "text": "RAG combines LLMs with vector search...",
  "embedding": [0.23, -0.45, ...],
  "metadata": {
    "source": "rag-guide.pdf",
    "page": 5,
    "section": "RAG Architecture",
    "timestamp": "2024-12-01",
    "tags": ["RAG", "LLMs", "vector search"]
  }
}

Use: Filtering results by metadata (source = "rag-guide.pdf").

---

Step 3: Embedding generation

Option 1: The OpenAI API

Input: The chunk text (up to 8191 tokens)
Model: text-embedding-3-small (1536D)
Cost: $0.00002 / 1K tokens

A batch of 1000 chunks × 500 tokens = 500K tokens
→ Cost: $0.01 (one cent)

Advantage: High quality, simple.
Disadvantage: A recurring cost (on every re-indexing).

---

Option 2: A local model (Sentence-BERT)

Model: all-MiniLM-L6-v2 (384D)
Cost: $0 (after setup)
Speed: ~100 chunks/second (on CPU)

Advantage: Free, private.
Disadvantage: Lower quality than OpenAI.

---

Step 4: Storing in a Vector DB

A Pinecone example:

For each chunk:
  1. Generate the embedding
  2. Upsert to Pinecone:
     - id: "chunk-123"
     - values: [0.23, -0.45, ...]
     - metadata: {"source": "doc.pdf", "page": 5}

Batch upsert: 100-1000 chunks per request (faster).

---

The complete pipeline (an example)

Input: 1000 PDFs (averaging 10 pages each)

The process:

1. Extract the text from the PDFs (PyPDF2, pdfplumber)
   → 10K pages of text

2. Chunking (500 tokens, 50 overlap)
   → ~50K chunks

3. Generate the embeddings (the OpenAI API)
   → 50K embeddings (1536D)
   → 50K chunks × 500 tokens = 25M tokens
   → Cost: 25,000 × $0.00002 = ~$0.50

4. Upsert to Pinecone (batches of 100)
   → 500 requests
   → Time: ~5-10 minutes

Total: ~15-20 minutes, ~$0.50

---

Common troubleshooting

Problem 1: Chunks that are too small

Symptom: Results without enough context

Solution: Increase the chunk size (500 → 800 tokens)

---

Problem 2: Chunks that are too large

Symptom: Imprecise results (a lot of noise)

Solution: Reduce the chunk size (1500 → 800 tokens)

---

Problem 3: Loss of context between chunks

Symptom: A query that requires 2 chunks doesn't get a complete answer

Solution: Increase the overlap (50 → 100 tokens) or use parent-child chunks

---

Summary

Key points:

• Chunking: 500-800 tokens with a 50-100 overlap
• Strategies: Fixed-size, paragraph, sentence, hierarchical
• Metadata: source, page, section, timestamp, tags
• Embeddings: The OpenAI API (quality) or local (free)
• The pipeline: Extract → Chunk → Embed → Store

---

Next capsule: 04-query-processing.md — Embedding the query, kNN, reranking.