2. Keyword Search: Traditional Search by Keywords

Overview

Keyword search is the traditional method: find documents that contain the exact words from the query (or variants via stemming/lemmatization). It's the technology behind classic search engines, SQL databases with FULLTEXT, and basic Elasticsearch.

How keyword search works

The basic algorithm:

1. Tokenize the query: "domestic dogs" → ["domestic", "dogs"]
2. For each document:
     Does it contain "domestic" or "dogs"?
3. Return the documents that contain at least one word
4. Sort by relevance (TF-IDF, BM25)

TF-IDF: The classic metric

TF-IDF = Term Frequency × Inverse Document Frequency

TF (term frequency):

TF = (times the word appears in the document) / (total words in the document)

IDF (term rarity):

IDF = log(total documents / documents containing the word)

TF-IDF combined:

TF-IDF = TF × IDF

Interpretation:

Words that are frequent in the document → high TF
Words that are rare in the corpus → high IDF
Common words ("the", "of") → low IDF (less relevant)

BM25: An improvement on TF-IDF

BM25 (Best Matching 25) is an improved version of TF-IDF used in Elasticsearch and modern systems.

Improvements over TF-IDF:

Frequency saturation: A word that appears 10 times vs 100 times doesn't give 10x the score
Length normalization: Long documents don't dominate the results
Tunable parameters: k1, b (they control saturation and normalization)

Result: BM25 is more robust than pure TF-IDF.

A keyword search example

Query: "domestic dogs"

Documents:

Doc 1: "Dogs are loyal domestic animals"
       → Contains "dogs" ✅ and "domestic" ✅ → high score

Doc 2: "The hound is a companion animal"
       → Contains NEITHER "dogs" nor "domestic" ❌ → score 0

Doc 3: "Cats are independent pets"
       → Contains NEITHER "dogs" nor "domestic" ❌ → score 0

Doc 4: "Dogs and cats are the most common pets"
       → Contains "dogs" ✅ but NOT "domestic" ❌ → medium score

Ranking:

Doc 1 (contains both words)
Doc 4 (contains one word)
Doc 2 and Doc 3 (no match)

Improvement techniques

1. Stemming (reducing words to a root):

"dogs" → "dog"
"dog"  → "dog"
→ Both are a match ✅

2. Lemmatization:

"running" → "run"
"ran"     → "run"
→ A match on the base form ✅

3. Stop words (filtering out common words):

Query: "the dog in the house"
→ Filter out: ["the", "in"]
→ Effective query: "dog house"

4. Manual synonyms:

"car" → also search for "auto", "automobile"

Advantages of keyword search

1. Precision for exact searches:

Query: "article 42 of law X"
→ You only want documents with "article 42"
→ Keyword is perfect ✅

2. Fast and efficient:

Inverted indexes (Elasticsearch) are very fast
Scalable to millions of documents

3. Explainable:

You can see exactly why a document was returned (it contains the words)

4. No embeddings needed:

You don't need an ML model or an external API
It works with any text

Limitations of keyword search

Limitation 1: It doesn't understand synonyms

Query: "dog"
Doc: "The hound is loyal"
→ NO match ❌ (even though "hound" = "dog")

Limitation 2: It doesn't understand related concepts

Query: "domestic animals"
Doc: "Cats are pets"
→ NO match ❌ (even though "pets" ≈ "domestic animals")

Limitation 3: Sensitive to the exact vocabulary

Query: "how to optimize performance"
Doc: "A guide to improving speed"
→ Low score (different words: "optimize" ≠ "improving", "performance" ≠ "speed")

Limitation 4: It captures neither order nor context

Query: "bank"
Doc 1: "I went to the bank to withdraw money" (the institution)
Doc 2: "I sat on a bank in the park" (the seat)
→ Both get the same score (both contain "bank")
→ It doesn't distinguish meaning by context

Summary

Key points:

Keyword search: Exact word matches
TF-IDF: Frequency × rarity (the classic metric)
BM25: An improved version (saturation, normalization)
Techniques: Stemming, lemmatization, stop words, synonyms
Advantages: Precise for exact matches, fast, explainable
Limitations: No synonyms, no concepts, no context

Next capsule: 03-semantic-search.md — Vector search with embeddings.