Embeddings vs Keyword Search

Capsule overview

Not every search problem requires embeddings—sometimes keyword search (BM25, Elasticsearch) is more appropriate, faster, and cheaper.

In this capsule you'll learn the fundamental difference between embeddings (semantic) and keyword search (lexical), when to use each one, the pros and cons of both approaches, and how to combine them in hybrid search to get the best of both worlds.

You'll also see practical code comparing both methods side by side and learn to make informed architecture decisions.

Keyword Search: BM25 and TF-IDF

What keyword search is:

Search based on matching exact (or stemmed) words between the query and the document.

Main algorithms:

• TF-IDF (Term Frequency - Inverse Document Frequency): Classic, simple
• BM25 (Best Match 25): An improvement over TF-IDF, standard in Elasticsearch

How BM25 works:

1. Tokenization:
   Query: "python tutorial" → ["python", "tutorial"]
   Doc 1: "Python tutorial for beginners" → ["python", "tutorial", "for", "beginners"]

2. Matching:
   How many words from the query are in the doc?
   Doc 1: 2/2 words (100% match) → High score

3. Scoring (simplified):
   score(doc, query) = Σ IDF(term) × TF(term, doc)
   
   IDF = log(N / df(term))  # Rare terms → higher weight
   TF = freq(term, doc)     # Frequent terms in doc → higher weight

Characteristics:

• ✅ Fast (inverted index)
• ✅ Exact (literal matching)
• ❌ Doesn't understand synonyms
• ❌ Doesn't understand context

Example with Elasticsearch (conceptual):

from elasticsearch import Elasticsearch

es = Elasticsearch()

# Index documents
documents = [
    {"id": 1, "text": "Python tutorial for beginners"},
    {"id": 2, "text": "JavaScript guide for developers"},
    {"id": 3, "text": "Learn Python programming"}
]

for doc in documents:
    es.index(index="docs", id=doc["id"], document=doc)

# Search with BM25 (default in Elasticsearch)
query = "python tutorial"
results = es.search(index="docs", query={
    "match": {
        "text": query
    }
})

# Results sorted by BM25 score
for hit in results["hits"]["hits"]:
    print(f"Doc {hit['_id']}: {hit['_source']['text']} (score: {hit['_score']})")

Expected output:

Doc 1: Python tutorial for beginners (score: 2.45)  ← Both words
Doc 3: Learn Python programming (score: 1.12)      ← Only "Python"
Doc 2: JavaScript guide for developers (score: 0)  ← No words

Semantic Search: Embeddings

What semantic search is:

Search based on semantic meaning via dense vectors.

How it works (already covered in previous capsules):

1. Embedding:
   Query: "python tutorial" → [0.023, -0.145, 0.892, ..., 0.567] (1536D)
   Doc 1: "Python guide" → [0.025, -0.143, 0.895, ..., 0.570] (1536D)

2. Similarity:
   cosine_similarity(query_emb, doc1_emb) → 0.95 (very similar)

3. Ranking:
   Sort docs by descending similarity

Characteristics:

• ✅ Understands synonyms ("tutorial" ≈ "guide")
• ✅ Understands paraphrases
• ❌ Slower (vector computation)
• ❌ More costly (API calls or GPU)

Direct comparison: Same query, both methods

Setup: 5 documents

documents = [
    {"id": 1, "text": "Python tutorial for beginners"},
    {"id": 2, "text": "Learn Python programming from scratch"},
    {"id": 3, "text": "JavaScript guide for developers"},
    {"id": 4, "text": "How to start coding in Python"},
    {"id": 5, "text": "Java programming basics"}
]

query = "python tutorial"

Method 1: BM25 (keyword)

# Simple BM25 simulation (without Elasticsearch)
def simple_bm25(query, doc):
    """
    Simplified BM25 simulation
    Counts word matches (stemmed)
    """
    query_words = set(query.lower().split())
    doc_words = set(doc.lower().split())
    
    # Matches
    matches = query_words.intersection(doc_words)
    
    # Score = # of matches (simplified)
    return len(matches)

# Apply to all docs
bm25_results = []
for doc in documents:
    score = simple_bm25(query, doc["text"])
    bm25_results.append((doc["id"], doc["text"], score))

# Sort by score
bm25_results.sort(key=lambda x: x[2], reverse=True)

print("BM25 Results:")
for doc_id, text, score in bm25_results:
    print(f"  Doc {doc_id}: {text} (score: {score})")

Output:

BM25 Results:
  Doc 1: Python tutorial for beginners (score: 2)  ← "python" + "tutorial"
  Doc 2: Learn Python programming from scratch (score: 1)  ← Only "python"
  Doc 4: How to start coding in Python (score: 1)  ← Only "python"
  Doc 3: JavaScript guide for developers (score: 0)  ← None
  Doc 5: Java programming basics (score: 0)  ← None

Notice: Doc 2 and Doc 4 have the SAME score, but Doc 2 is more relevant (content about learning Python).

Method 2: Embeddings (semantic)

from openai import OpenAI
import numpy as np
import os
from dotenv import load_dotenv

load_dotenv()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def get_embedding(text):
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return np.array(response.data[0].embedding)

def cosine_similarity(vec_a, vec_b):
    return np.dot(vec_a, vec_b) / (np.linalg.norm(vec_a) * np.linalg.norm(vec_b))

# Embed query
query_embedding = get_embedding(query)

# Embed documents and calculate similarities
semantic_results = []
for doc in documents:
    doc_embedding = get_embedding(doc["text"])
    similarity = cosine_similarity(query_embedding, doc_embedding)
    semantic_results.append((doc["id"], doc["text"], similarity))

# Sort by similarity
semantic_results.sort(key=lambda x: x[2], reverse=True)

print("\nSemantic Results (Embeddings):")
for doc_id, text, sim in semantic_results:
    print(f"  Doc {doc_id}: {text} (score: {sim:.4f})")

Expected output:

Semantic Results (Embeddings):
  Doc 1: Python tutorial for beginners (score: 0.92)  ← Exact
  Doc 2: Learn Python programming from scratch (score: 0.88)  ← Semantic
  Doc 4: How to start coding in Python (score: 0.82)  ← Semantic
  Doc 5: Java programming basics (score: 0.72)  ← Related (language)
  Doc 3: JavaScript guide for developers (score: 0.68)  ← Somewhat related

Notice:

• Doc 2 and Doc 4 have different scores (embeddings capture the nuance)
• Doc 5 (Java) appears with a score > 0 (related even though not Python)

Pros and cons

BM25 / Keyword Search

Pros:

✅ 1. Perfect exact search

Query: "ERROR-404-USER-123"
BM25: Finds exactly "ERROR-404-USER-123" ✅
Embeddings: May confuse it with "ERROR-404-USER-124" ⚠️

✅ 2. Fast (milliseconds)

# Elasticsearch with inverted index:
# 10 million docs → ~5-10ms per query

✅ 3. Free (open-source)

# Elasticsearch, Apache Solr: Free
# Embeddings: $0.00002/1K tokens (OpenAI)

✅ 4. Interpretable

# You can see WHICH words matched:
Query: "python tutorial"
Doc: "Python tutorial for beginners"
Match: ["python", "tutorial"] ← Clear

Cons:

❌ 1. Doesn't understand synonyms

Query: "tutorial"
Doc 1: "Python tutorial" ✅ Match
Doc 2: "Python guide" ❌ No match (even though "guide" = "tutorial")

❌ 2. Doesn't understand paraphrases

Query: "how to reset my laptop"
Doc 1: "reset laptop" ✅ Match
Doc 2: "restart computer" ❌ No match (even though it means the same thing)

❌ 3. Sensitive to vocabulary

# The user uses different terms than the document:
Query: "car"
Doc: "automobile" ❌ No match (same thing, different word)

Embeddings / Semantic Search

Pros:

✅ 1. Understands synonyms

Query: "tutorial"
Doc: "Python guide" ✅ Finds it (similarity ~0.85)

✅ 2. Understands paraphrases

Query: "how to reset my laptop"
Doc: "restart computer" ✅ Finds it (similarity ~0.88)

✅ 3. Multilingual (with an appropriate model)

Query: "python tutorial" (English)
Doc: "tutorial de Python" (Spanish) ✅ Finds it (similarity ~0.90)

✅ 4. Captures context

Query: "bank"
Doc 1: "river bank" → Embedding A
Doc 2: "bank to withdraw money" → Embedding B
# Embeddings A and B are different (different context)

Cons:

❌ 1. May over-generalize

Query: "Python 3.9"
BM25: Finds exactly "Python 3.9" ✅
Embeddings: May return "Python 3.10" (similar but not exact) ⚠️

❌ 2. Slower

# Embedding generation:
# 1 query → 1 API call (~100-300ms)

# Similarity calculation:
# 1 million docs → 1 million cosine similarity calculations
# Without an index: ~10-30 seconds 😱
# With a vector DB (HNSW): ~50-200ms ✅

❌ 3. More costly

# OpenAI embeddings: $0.00002/1K tokens
# 1 million docs × 500 tokens average × $0.00002 = $10 USD
# BM25 / Elasticsearch: $0 (open-source)

❌ 4. Less interpretable

# You can't see WHY similarity = 0.85
# It's 1536 dimensions (black box)

Use cases: When to use what

USE BM25 when:

USE Embeddings when:

Hybrid Search: The best of both worlds

Concept:

Combine BM25 (keyword) + Embeddings (semantic) into a single score.

Typical formula:

hybrid_score = α × bm25_score + (1 - α) × semantic_score

where α ∈ [0, 1] (typically α = 0.5)

Conceptual implementation:

def hybrid_search(query, documents, alpha=0.5):
    """
    Hybrid search: BM25 + Embeddings
    
    Args:
        query: The user's query
        documents: List of documents
        alpha: Weight of BM25 (1-alpha = weight of embeddings)
    
    Returns:
        Documents sorted by hybrid score
    """
    # Step 1: BM25 scores (normalize to [0, 1])
    bm25_scores = {}
    for doc in documents:
        score = simple_bm25(query, doc["text"])
        bm25_scores[doc["id"]] = score
    
    # Normalize BM25 scores
    max_bm25 = max(bm25_scores.values()) if bm25_scores else 1
    bm25_normalized = {
        doc_id: score / max_bm25 
        for doc_id, score in bm25_scores.items()
    }
    
    # Step 2: Semantic scores (cosine similarity already in [0, 1])
    query_embedding = get_embedding(query)
    semantic_scores = {}
    for doc in documents:
        doc_embedding = get_embedding(doc["text"])
        sim = cosine_similarity(query_embedding, doc_embedding)
        # Convert from [-1, 1] to [0, 1]
        sim_normalized = (sim + 1) / 2
        semantic_scores[doc["id"]] = sim_normalized
    
    # Step 3: Combine scores
    hybrid_scores = []
    for doc in documents:
        doc_id = doc["id"]
        bm25_score = bm25_normalized.get(doc_id, 0)
        semantic_score = semantic_scores.get(doc_id, 0)
        
        # Hybrid score
        hybrid_score = alpha * bm25_score + (1 - alpha) * semantic_score
        
        hybrid_scores.append({
            "id": doc_id,
            "text": doc["text"],
            "hybrid_score": hybrid_score,
            "bm25_score": bm25_score,
            "semantic_score": semantic_score
        })
    
    # Sort by hybrid score
    hybrid_scores.sort(key=lambda x: x["hybrid_score"], reverse=True)
    
    return hybrid_scores

# Example
query = "python tutorial"
results = hybrid_search(query, documents, alpha=0.5)

print("Hybrid Search Results (α=0.5):")
for r in results:
    print(f"Doc {r['id']}: {r['text']}")
    print(f"  BM25: {r['bm25_score']:.2f}, Semantic: {r['semantic_score']:.2f}, Hybrid: {r['hybrid_score']:.2f}\n")

Expected output:

Hybrid Search Results (α=0.5):
Doc 1: Python tutorial for beginners
  BM25: 1.00, Semantic: 0.96, Hybrid: 0.98

Doc 2: Learn Python programming from scratch
  BM25: 0.50, Semantic: 0.94, Hybrid: 0.72

Doc 4: How to start coding in Python
  BM25: 0.50, Semantic: 0.91, Hybrid: 0.71

Doc 5: Java programming basics
  BM25: 0.00, Semantic: 0.86, Hybrid: 0.43

Doc 3: JavaScript guide for developers
  BM25: 0.00, Semantic: 0.84, Hybrid: 0.42

Tuning α (BM25 vs Semantic trade-off):

# α = 0.0 → 100% Semantic (ignores keywords)
# α = 0.5 → 50/50 balance
# α = 1.0 → 100% BM25 (ignores semantic)

# Example: Code search (keywords matter)
α = 0.7  # 70% BM25, 30% Semantic

# Example: Conceptual search (meaning matters)
α = 0.3  # 30% BM25, 70% Semantic

Real example: Elasticsearch + Vector Search

Elasticsearch 8.0+ includes native support for embeddings:

from elasticsearch import Elasticsearch

es = Elasticsearch()

# Index with embeddings
doc = {
    "text": "Python tutorial for beginners",
    "embedding": get_embedding("Python tutorial for beginners")  # [1536 dims]
}

es.index(index="hybrid-docs", document=doc)

# Hybrid query (BM25 + KNN)
query = "python tutorial"
query_embedding = get_embedding(query)

response = es.search(index="hybrid-docs", query={
    "bool": {
        "should": [
            # BM25 (keyword)
            {
                "match": {
                    "text": {
                        "query": query,
                        "boost": 0.5  # α = 0.5
                    }
                }
            },
            # KNN (semantic)
            {
                "knn": {
                    "field": "embedding",
                    "query_vector": query_embedding,
                    "k": 10,
                    "num_candidates": 100,
                    "boost": 0.5  # 1-α = 0.5
                }
            }
        ]
    }
})

# Results combine both scores
for hit in response["hits"]["hits"]:
    print(f"{hit['_source']['text']} (score: {hit['_score']})")

Exercises

Exercise 1: Implement simple BM25

Implement a simplified BM25 that counts matches:

def simple_bm25(query, doc):
    # Implement word matching
    pass

query = "python tutorial"
doc = "Python tutorial for beginners"

# Should return 2 (both words match)

def simple_bm25(query, doc):
    """
    Simplified BM25: counts word matches
    """
    # Convert to lowercase and split into words
    query_words = set(query.lower().split())
    doc_words = set(doc.lower().split())
    
    # Count matches
    matches = query_words.intersection(doc_words)
    
    return len(matches)

query = "python tutorial"
doc = "Python tutorial for beginners"

score = simple_bm25(query, doc)
print(f"BM25 score: {score}")  # 2

Exercise 2: Compare BM25 vs Semantic

Compare both methods for this query:

query = "red car"

documents = [
    "Selling a red automobile",
    "Red sports car",
    "Used red vehicle"
]

# Implement search with both methods
# Which one finds all 3 documents?

# BM25
print("BM25 Results:")
bm25_results = []
for doc in documents:
    score = simple_bm25(query, doc)
    bm25_results.append((doc, score))
    print(f"  '{doc}' → score: {score}")

# Semantic
print("\nSemantic Results:")
query_emb = get_embedding(query)
semantic_results = []
for doc in documents:
    doc_emb = get_embedding(doc)
    sim = cosine_similarity(query_emb, doc_emb)
    semantic_results.append((doc, sim))
    print(f"  '{doc}' → score: {sim:.4f}")

BM25 Results:
  'Selling a red automobile' → score: 1  (only "red")
  'Red sports car' → score: 2  ("red", "car")
  'Used red vehicle' → score: 1  (only "red")

Semantic Results:
  'Selling a red automobile' → score: 0.92  ("automobile" ≈ "car")
  'Red sports car' → score: 0.95  (exact + context)
  'Used red vehicle' → score: 0.90  ("vehicle" ≈ "car")

Exercise 3: Hybrid search

Implement hybrid search with α=0.6:

query = "connection error"
documents = [
    "Error connecting to database",
    "Network connection problem",
    "Connection failure"
]

# Implement hybrid search with α=0.6 (60% BM25, 40% Semantic)

def hybrid_search(query, documents, alpha=0.6):
    # BM25 scores
    bm25_scores = []
    for doc in documents:
        score = simple_bm25(query, doc)
        bm25_scores.append(score)
    
    # Normalize BM25
    max_bm25 = max(bm25_scores) if max(bm25_scores) > 0 else 1
    bm25_normalized = [score / max_bm25 for score in bm25_scores]
    
    # Semantic scores
    query_emb = get_embedding(query)
    semantic_scores = []
    for doc in documents:
        doc_emb = get_embedding(doc)
        sim = cosine_similarity(query_emb, doc_emb)
        # Normalize from [-1, 1] to [0, 1]
        sim_normalized = (sim + 1) / 2
        semantic_scores.append(sim_normalized)
    
    # Hybrid scores
    hybrid_results = []
    for i, doc in enumerate(documents):
        hybrid_score = alpha * bm25_normalized[i] + (1 - alpha) * semantic_scores[i]
        hybrid_results.append({
            "doc": doc,
            "bm25": bm25_normalized[i],
            "semantic": semantic_scores[i],
            "hybrid": hybrid_score
        })
    
    # Sort by hybrid score
    hybrid_results.sort(key=lambda x: x["hybrid"], reverse=True)
    
    return hybrid_results

query = "connection error"
results = hybrid_search(query, documents, alpha=0.6)

print("Hybrid Results (α=0.6):")
for r in results:
    print(f"  '{r['doc']}'")
    print(f"    BM25: {r['bm25']:.2f}, Semantic: {r['semantic']:.2f}, Hybrid: {r['hybrid']:.2f}\n")

Hybrid Results (α=0.6):
  'Error connecting to database'
    BM25: 0.50, Semantic: 0.94, Hybrid: 0.68

  'Network connection problem'
    BM25: 1.00, Semantic: 0.96, Hybrid: 0.98

  'Connection failure'
    BM25: 0.50, Semantic: 0.92, Hybrid: 0.67

Common troubleshooting

Problem 1: BM25 doesn't find synonyms

Query: "car"
Doc: "red automobile"

BM25 score: 0  # ❌ No match

Solution: Use hybrid search or semantic only.

Problem 2: Embeddings over-generalize

Query: "Python 3.9"
Semantic returns: "Python 3.10" (similarity 0.95)

# But the user wanted SPECIFICALLY 3.9

Solution: Use hybrid search with a high α (e.g., 0.7) to give more weight to keywords.

Problem 3: Hybrid scores dominated by one method

# BM25 scores: 0.1, 0.2, 0.3
# Semantic scores: 0.85, 0.90, 0.95

# Hybrid (α=0.5): Semantic dominates

Solution: Adjust α or normalize both scores better to the same scale.

Summary

What you learned:

• ✅ BM25: Keyword search, fast, exact, doesn't capture synonyms
• ✅ Embeddings: Semantic search, synonyms, slower, more costly
• ✅ Hybrid: Combines both with α (trade-off)
• ✅ When to use what: IDs/codes → BM25, synonyms/paraphrases → Embeddings
• ✅ Best practice: Hybrid search (α=0.5 as a baseline)

Architecture decisions:

1. Limited budget + homogeneous corpus → BM25
2. Linguistic variability + flexible budget → Embeddings
3. Production + better result → Hybrid search

Additional resources

1. BM25 Explained - Elasticsearch
2. Hybrid Search Guide - Pinecone
3. Elasticsearch Vector Search - Official docs
4. Semantic vs Keyword - SBERT
5. Reciprocal Rank Fusion - Alternative to weighted hybrid

In the next capsule

Capsule 07: Architecture Overview

You'll learn:

• High-level Transformers (encoder-only)
• Tokenization with tiktoken
• Self-attention (conceptual)
• Pooling strategies (mean, CLS)
• How an embedding is generated end-to-end

From search decisions to technical architecture.

Module 1 - Embeddings Deep Dive Guide Choosing the right tool for each problem