Module 5: Keyword vs Semantic Search
3. Semantic Search: Vector Search by Meaning
Overview
Semantic search uses embeddings (vectors) to search by meaning instead of by exact words. It finds conceptually similar documents even when they use different vocabulary. It's the foundation of RAG and modern search systems.
How semantic search works
The algorithm:
1. Convert the query into an embedding (a 1536D vector)
Query: "domestic dogs" → [0.23, -0.45, ..., -0.34]
2. Convert all the documents into embeddings (once, offline)
Doc 1 → [0.23, -0.45, ..., -0.34]
Doc 2 → [0.25, -0.43, ..., -0.32]
...
3. Compute the cosine similarity between the query and each document
4. Return the top-K documents with the highest similarity
Example: Semantic search in action
Query: "domestic dogs"
Query embedding: [0.23, -0.45, ..., -0.34]
Documents:
Doc 1: "Dogs are loyal domestic animals"
→ Embedding: [0.23, -0.45, ..., -0.34]
→ Similarity: 0.98 ✅ (nearly identical)
Doc 2: "The hound is a companion animal"
→ Embedding: [0.24, -0.44, ..., -0.33]
→ Similarity: 0.95 ✅ (very high, even though it doesn't contain "dog")
Doc 3: "Cats are independent pets"
→ Embedding: [0.25, -0.43, ..., -0.32]
→ Similarity: 0.85 ✅ (high, a related concept)
Doc 4: "Cars have wheels"
→ Embedding: [9.34, 5.21, ..., 7.56]
→ Similarity: 0.12 ❌ (low, unrelated)
Ranking:
- Doc 1 (0.98)
- Doc 2 (0.95) — ✅ Found even though it does NOT contain "dog"
- Doc 3 (0.85) — ✅ Found even though it contains neither "dog" nor "domestic"
Advantages of semantic search
1. It understands synonyms:
Query: "dog"
Doc: "The hound is loyal"
→ ✅ Match (similar embeddings)
2. It understands related concepts:
Query: "domestic animals"
Doc: "Cats are pets"
→ ✅ Match (related concepts)
3. It understands intent:
Query: "how to improve my app's speed"
Doc: "A performance optimization guide"
→ ✅ Match (the same concept, different vocabulary)
4. It's robust to variations:
Query: "machine learning"
Doc: "Introduction to ML"
→ ✅ Match (ML = the abbreviation)
Limitations of semantic search
Limitation 1: It doesn't guarantee exact words
Query: "article 42"
Doc: "Article 41 states..."
→ It can get a high similarity ❌ (even though it's the wrong article)
Solution: Use keyword search for exact matches, or go hybrid.
Limitation 2: Model dependence
If the model was NOT trained on your domain:
→ The embeddings can be suboptimal
Solution: Fine-tuning or a specialized model.
Limitation 3: Computational cost
Generating embeddings → requires an API (OpenAI) or a local model
The cost per query is higher than keyword search
Solution: Cache the embeddings, use efficient models.
Limitation 4: The "black box"
Why does this document have a score of 0.87?
→ Hard to explain (vectors are abstract)
Solution: Keyword search is more explainable.
A direct comparison: The same query
Query: "domestic animals"
With keyword search:
Doc 1: "Dogs are domestic animals"
→ ✅ Contains "animals" and "domestic" → high score
Doc 2: "The hound is loyal"
→ ❌ Contains NEITHER "animals" nor "domestic" → score 0
Doc 3: "Cats are pets"
→ ❌ Contains NEITHER "animals" nor "domestic" → score 0
Result: Only Doc 1 is found.
With semantic search:
Doc 1: "Dogs are domestic animals"
→ An embedding similar to the query → similarity 0.95 ✅
Doc 2: "The hound is loyal"
→ "hound" has an embedding similar to "dog" → similarity 0.82 ✅
Doc 3: "Cats are pets"
→ "pets" is a related concept → similarity 0.78 ✅
Result: 3 documents found (Doc 1, 2, 3).
The advantage of semantic search: It finds Doc 2 and Doc 3 even though they do NOT contain the exact words.
The technical flow (conceptual)
Phase 1: Indexing (offline, once):
For each document:
1. Document → embedding (via the OpenAI API or a local model)
2. Store the embedding in a vector database (Pinecone, Weaviate)
Phase 2: Searching (online, for every query):
1. Query → embedding
2. Search the vector database (kNN with an HNSW/IVF index)
3. Return the top-K documents with the highest cosine
Typical technologies
Embedding models:
- OpenAI
text-embedding-3-small(1536D) - OpenAI
text-embedding-3-large(3072D) - Sentence-BERT (768D)
- Cohere embeddings (1024D)
Vector databases:
- Pinecone (managed, HNSW)
- Weaviate (open-source, HNSW)
- Qdrant (open-source, HNSW)
- FAISS (a library, IVF)
- Milvus (open-source, multiple indexes)
The cost of semantic search
Indexing cost:
1M documents × $0.00013/1K tokens (OpenAI ada-002)
Assuming an average of 500 tokens/document
→ 500M tokens total
→ Cost: ~$65 (one time)
Query cost:
A typical query: 10-50 tokens
→ $0.0000013 - $0.0000065 per query with OpenAI
1000 queries/day = $0.0013 - $0.0065/day
The alternative: A local model (Sentence-BERT) → cost 0 after the initial setup.
Summary
Key points:
- Semantic search: Embeddings + cosine
- Advantages: Synonyms, related concepts, intent
- Limitations: No guarantee of exact matches, model dependence, cost
- The flow: Index the docs (offline) → embed the query → kNN (online)
- Technologies: OpenAI, Pinecone, Weaviate, BERT
Next capsule: 04-comparison.md — Keyword vs semantic side by side.