Module 7: RAG and Semantic Search

4. The RAG Flow Step by Step, with a Real Example

Next →

Overview

This capsule walks you through a complete RAG example: from raw documents to a generated answer, with real data and specific decisions at every step.

The scenario: A Documentation Q&A System

Context:
A startup with 50 internal documents (product guides, tutorials, FAQs). They want a chatbot that answers employees' questions.

The documents:

  • setup-guide.pdf (20 pages)
  • support-faq.pdf (15 pages)
  • product-architecture.pdf (30 pages)
  • ... (47 more documents)

STEP 1: Indexing the documents

1.1 Text extraction

setup-guide.pdf → Extract the text (PyPDF2)
→ 10,000 words (~12,500 tokens)

support-faq.pdf → Extract the text
→ 7,500 words (~9,375 tokens)

...

Total: 50 documents → ~300K tokens

1.2 Chunking

Strategy: Fixed-size (500 tokens, 50 overlap)

setup-guide.pdf (12,500 tokens):
→ Chunk 1: tokens 0-500
→ Chunk 2: tokens 450-950 (50 overlap)
→ Chunk 3: tokens 900-1400
...
→ 26 chunks

support-faq.pdf (9,375 tokens):
→ 20 chunks

...

Total: 50 docs → ~620 chunks

1.3 Generating the embeddings

Model: OpenAI text-embedding-3-small (1536D)

For each chunk:
  text = "Setup guide: To configure..."
  embedding = openai.embeddings.create(
      input=text,
      model="text-embedding-3-small"
  )
  → [0.23, -0.45, 0.12, ..., -0.34]

Cost: 620 chunks × 500 tokens = 310K tokens
      → $0.00002/1K tokens × 310 = $0.0062 (~0.6 of a cent)

1.4 Storing in Pinecone

pinecone.init(api_key="...", environment="us-west1-gcp")
index = pinecone.Index("internal-documentation")

For each chunk:
  index.upsert(vectors=[
      {
          "id": "setup-guide-chunk-1",
          "values": [0.23, -0.45, ...],
          "metadata": {
              "source": "setup-guide.pdf",
              "page": 1,
              "text": "Setup guide: To configure..."
          }
      }
  ])

Total: 620 upserts (batches of 100)
Time: ~30 seconds

The complete indexing:

  • Cost: $0.0062
  • Time: ~5 minutes (extraction + chunking + embedding + upsert)

STEP 2: The user's query

User query: "How do I set up the development environment?"

STEP 3: Retrieval (searching)

3.1 Embedding the query

query = "How do I set up the development environment?"

query_embedding = openai.embeddings.create(
    input=query,
    model="text-embedding-3-small"
)
→ [0.24, -0.44, 0.13, ..., -0.33]

Latency: 120ms

3.2 kNN search in Pinecone

results = index.query(
    vector=query_embedding,
    top_k=5,
    include_metadata=True
)

Results:

1. id: setup-guide-chunk-3
   score: 0.91
   metadata:
     source: setup-guide.pdf
     page: 2
     text: "To set up the development environment:
            1. Install Node.js 18+
            2. Clone the repository
            3. npm install
            4. Configure .env with the API keys..."

2. id: setup-guide-chunk-5
   score: 0.88
   metadata:
     source: setup-guide.pdf
     page: 3
     text: "The required environment variables:
            - DATABASE_URL: Your PostgreSQL URL
            - OPENAI_API_KEY: Your OpenAI key
            - PORT: The server port (default: 3000)..."

3. id: support-faq-chunk-12
   score: 0.85
   metadata:
     source: support-faq.pdf
     page: 8
     text: "Frequently asked: How do I fix the 'module not found' error?
            Answer: Check that you ran npm install..."

4. id: architecture-chunk-7
   score: 0.79
   metadata:
     source: product-architecture.pdf
     page: 4
     text: "The technical stack: Node.js + PostgreSQL + Redis..."

5. id: setup-guide-chunk-1
   score: 0.76
   metadata:
     source: setup-guide.pdf
     page: 1
     text: "Welcome to the product setup guide..."

Latency: 40ms

STEP 4: Augmentation (building the prompt)

# Build the context from the top-5 chunks
context = ""
for i, result in enumerate(results, 1):
    context += f"\n[Chunk {i}] ({result['metadata']['source']}, page {result['metadata']['page']})\n"
    context += result['metadata']['text'] + "\n"

# The complete prompt
prompt = f"""
You are an internal assistant for our company. Answer based ONLY on the documentation provided.

DOCUMENTATION:
{context}

---

THE USER'S QUESTION: {query}

INSTRUCTIONS:
- Answer clearly and concisely
- If the answer isn't in the documentation, say "I don't have information about this"
- Cite the source where relevant (e.g. "According to the setup guide, page 2...")
"""

The final prompt (~2200 tokens):

You are an internal assistant for our company. Answer based ONLY on the documentation provided.

DOCUMENTATION:

[Chunk 1] (setup-guide.pdf, page 2)
To set up the development environment:
1. Install Node.js 18+
2. Clone the repository
3. npm install
4. Configure .env with the API keys...

[Chunk 2] (setup-guide.pdf, page 3)
The required environment variables:
- DATABASE_URL: Your PostgreSQL URL
- OPENAI_API_KEY: Your OpenAI key
- PORT: The server port (default: 3000)...

[Chunk 3] (support-faq.pdf, page 8)
Frequently asked: How do I fix the 'module not found' error?
Answer: Check that you ran npm install...

[Chunk 4] (product-architecture.pdf, page 4)
The technical stack: Node.js + PostgreSQL + Redis...

[Chunk 5] (setup-guide.pdf, page 1)
Welcome to the product setup guide...

---

THE USER'S QUESTION: How do I set up the development environment?

INSTRUCTIONS:
- Answer clearly and concisely
- If the answer isn't in the documentation, say "I don't have information about this"
- Cite the source where relevant

STEP 5: Generation (the LLM produces the answer)

response = openai.chat.completions.create(
    model="gpt-4-turbo",
    messages=[
        {"role": "system", "content": "You are an internal assistant."},
        {"role": "user", "content": prompt}
    ],
    temperature=0.1  # A low temperature for precise answers
)

Latency: 3.2 seconds

The generated answer:

To set up the development environment, follow these steps (according to the setup guide, page 2):

1. **Install Node.js 18+**
   Download it from nodejs.org

2. **Clone the repository**

git clone https://github.com/company/project.git cd project


3. **Install the dependencies**

npm install


4. **Configure the environment variables**
Create a `.env` file with the following variables (setup guide, page 3):
- `DATABASE_URL`: Your PostgreSQL URL
- `OPENAI_API_KEY`: Your OpenAI API key
- `PORT`: The server port (default: 3000)

If you hit the "module not found" error, check that you ran `npm install` correctly (support FAQ, page 8).

Do you need help with any specific step?

STEP 6: Returning the answer to the user

The final output:

{
  "query": "How do I set up the development environment?",
  "response": "To set up the development environment...",
  "sources": [
    {"file": "setup-guide.pdf", "page": 2},
    {"file": "setup-guide.pdf", "page": 3},
    {"file": "support-faq.pdf", "page": 8}
  ],
  "latency_ms": 3360
}

A summary of the latency and cost

Latency:

- Query embedding: 120ms
- kNN search: 40ms
- Prompt construction: <10ms
- LLM generation: 3200ms
Total: 3360ms (~3.4 seconds)

Cost per query:

- Query embedding: $0.00000004 (2 tokens × $0.00002/1K)
- kNN search: Included in Pinecone (the $70/month plan)
- LLM generation: $0.022 (2200 input tokens × $0.01/1K)
Total: ~$0.022 per query

Possible optimizations:

  • Use GPT-3.5 → 10x cheaper, 2x faster
  • Cache frequent queries
  • Stream the response

Summary

Key points:

  • Indexing: 50 docs → 620 chunks → $0.0062, 5 min
  • Query: Embedding → kNN → Prompt → LLM → 3.4s, $0.022
  • Sources: The answer cites specific sources (explainable)
  • Precision: No hallucinations (the answer is grounded in real docs)

Next capsule: 05-rag-vs-fine-tuning.md — When to use each.

← Previous3. RAG Architecture: Components and FlowsNext →5. RAG vs Fine-tuning: When to Use Each