Capsule 06: HyDE — searching with the hypothetical answer instead of the question

Capsule overview

So far every query optimization technique manipulates the query as text (expansion, rewriting, decomposition). HyDE (Hypothetical Document Embeddings) does something different and counterintuitive: instead of searching with the query's embedding, you generate a hypothetical answer with an LLM and embed that answer. What you look for in the corpus is "documents similar to the kind of answer an expert would give to this" — semantically closer to the real docs than the raw question is.

The insight is geometric: queries and documents live in different regions of the embedding space. A query is short, interrogative, abstract. A document is long, declarative, specific. Even when they're about the same topic, their cosine distance is modest — typically 0.7-0.75. But a hypothetical document generated to answer the query lives in the same region of the space as the real docs — its similarity with relevant docs climbs to 0.85-0.92.

HyDE is the most sophisticated technique in the module. It's useful when nothing else is enough, especially for open-ended queries in domains where the document "style" is predictable (technical documentation, academic papers, manuals).

By the end of this capsule you'll be able to:

• ✅ Explain the geometric insight that justifies HyDE
• ✅ Implement basic HyDE with an LLM and embedding of the generated doc
• ✅ Implement the two main variants: Multi-HyDE and Hybrid (query + HyDE)
• ✅ Decide when HyDE beats query expansion or rewriting
• ✅ Anticipate the two main traps: hypothetical docs that hallucinate and disproportionate cost
• ✅ Compute HyDE's ROI compared to simpler techniques

Estimated time: 30-35 minutes

The geometric insight: queries and docs live in different regions

Picture the embedding space as a map where similar texts sit close together. Queries and documents don't share a neighborhood, even when they're about the same topic.

                  Embedding space (2D visualization)

           ┌─────────────────────────────────────────────────┐
           │                                                  │
           │   Q  Q  Q                                        │
           │     Q   Q                ← QUERIES region        │
           │   Q  Q                                           │
           │                                                  │
           │           ............                           │
           │            (gap)                                 │
           │                                                  │
           │                          D     D                 │
           │                       D    D       D             │
           │                       D    D    D     ← DOCS     │
           │                          D       D      region   │
           │                       D     D                    │
           │                                                  │
           └──────────────────────────────────────────────────┘

  cosine(query, doc) = typically 0.7-0.75
  (close but not very close — they're in different regions)

Why it happens: embedding models encode style as well as topic. A query "how do I implement OAuth2 in FastAPI?" has an interrogative tone, is short, abstract. A document "To implement OAuth2 in FastAPI, use the OAuth2PasswordBearer class from fastapi.security..." has a declarative tone, is long, concrete. Even though both talk about OAuth2 + FastAPI, the model puts them in different regions.

HyDE's trick: you generate a hypothetical document with an LLM. That doc now lives in the docs region:

Query  ──> LLM ──> Hypothetical doc ──> Embedding
"How to              "To implement                 (lives in
 implement            OAuth2 in FastAPI,            the region of
 OAuth2"              use OAuth2PasswordBearer..."  real docs)

When you search with the hypothetical doc's embedding, you find real docs that are close to it — and because that hypothetical doc "knows" what the right answer looks like, the real docs that match are typically the relevant ones.

Typical gain:

Cosine(query, real doc)        =  0.72
Cosine(hypothetical_doc, real) =  0.88
                                  ────
                                  +22% gain in similarity → better ranking

Basic implementation

# hyde.py
from openai import OpenAI
import os

client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))


HYDE_PROMPT = """Write a detailed technical document that would perfectly answer this question.

Write as if you are an expert author writing documentation or a tutorial. Be specific,
technical, and use natural language. Include code examples if relevant.

Length: 200-400 words.

Question: {query}

Document:"""


def generate_hypothetical_document(query: str) -> str:
    """Generate a document that would answer the query."""
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are a technical writer creating documentation."},
            {"role": "user", "content": HYDE_PROMPT.format(query=query)},
        ],
        temperature=0.3,  # low for consistency
        max_tokens=500,
    )
    return response.choices[0].message.content.strip()


def search_with_hyde(query: str, top_k: int = 5) -> dict:
    """HyDE search: generate hypothetical doc → embed → search."""
    # 1. Generate the hypothetical doc
    hyde_doc = generate_hypothetical_document(query)

    # 2. Search with the hypothetical doc (ChromaDB embeds it automatically)
    results = collection.query(
        query_texts=[hyde_doc],  # We search with the doc, not the query
        n_results=top_k,
    )

    return {
        "query": query,
        "hypothetical_doc": hyde_doc,
        "documents": results['documents'][0],
        "ids": results['ids'][0],
    }


# Try it out
result = search_with_hyde(
    "How do I implement OAuth2 in FastAPI?",
    top_k=5
)

print(f"Hypothetical doc generated:\n{result['hypothetical_doc'][:300]}...\n")
print(f"Top 5 retrieved docs:")
for i, doc in enumerate(result['documents'], 1):
    print(f"\n#{i}: {doc[:120]}...")

Typical output:

Hypothetical doc generated:
To implement OAuth2 in FastAPI, you first import `OAuth2PasswordBearer` from the
`fastapi.security` module. This class acts as a dependency that extracts the token
from the Authorization header. The basic setup requires installing `python-jose`
for JWT handling and `passlib` for password hashing.

Typical steps:

1. Define the scheme: `oauth2_scheme = OAuth2PasswordBearer(tokenUrl="token")`
2. Create a login endpoint that returns a JWT...

Top 5 retrieved docs:

#1: FastAPI OAuth2 implementation guide using OAuth2PasswordBearer dependency...
#2: Securing FastAPI endpoints with JWT tokens and password hashing using passlib...
#3: How to implement role-based access control in FastAPI applications using OAuth2...

Main variants

Variant 1: Multi-HyDE (more robust)

Generate several hypothetical documents with varied temperature, search with each one, fuse with RRF. More expensive but more robust against occasionally wrong hypothetical documents.

def multi_hyde_search(query: str, num_docs: int = 3, top_k: int = 5):
    """Generate N hypothetical docs and fuse the results."""
    all_rankings = []

    for i in range(num_docs):
        # Vary the temperature for diversity
        temp = 0.2 + (i * 0.2)  # 0.2, 0.4, 0.6
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": "You are a technical writer."},
                {"role": "user", "content": HYDE_PROMPT.format(query=query)},
            ],
            temperature=temp,
            max_tokens=500,
        )
        hyde_doc = response.choices[0].message.content.strip()

        # Search with this hypothetical doc
        results = collection.query(query_texts=[hyde_doc], n_results=10)
        all_rankings.append(results['ids'][0])

    # Fuse with RRF (seen in capsule 03)
    fused = reciprocal_rank_fusion(all_rankings)
    top_ids = [doc_id for doc_id, _ in fused[:top_k]]

    # Fetch the content
    return collection.get(ids=top_ids)

Gain: ~+3-5% recall vs single HyDE. Costs 3x more LLM calls.

Variant 2: Hybrid (query + HyDE)

Search with the original query AND with the hypothetical doc, then fuse. It combines the precision of the direct query with the recall of HyDE.

def hybrid_hyde_search(query: str, top_k: int = 5):
    """Combine the direct query search + HyDE."""
    # Search 1: direct query
    results_query = collection.query(query_texts=[query], n_results=10)

    # Search 2: HyDE
    hyde_doc = generate_hypothetical_document(query)
    results_hyde = collection.query(query_texts=[hyde_doc], n_results=10)

    # Fuse with RRF
    fused = reciprocal_rank_fusion([
        results_query['ids'][0],
        results_hyde['ids'][0],
    ])
    top_ids = [doc_id for doc_id, _ in fused[:top_k]]

    return collection.get(ids=top_ids)

Gain: a precision-recall balance. Useful when the direct query is good too but HyDE finds additional docs.

When HyDE beats the other techniques

                                        Is recall the problem?
                                                  │
                              ┌───────────────────┴──────────────────┐
                              │ Yes                                  │ No (problem = precision)
                              ▼                                      ▼
                  ┌─────────────────────┐           ┌─────────────────────────┐
                  │ Is the query        │           │ Consider:                │
                  │ ambiguous           │           │ - Re-ranking (M04)       │
                  │ (1-3 tokens)?        │           │ - Hybrid search (M05)    │
                  └──────────┬──────────┘           │ - Metadata filter (M06)  │
                             │                      └──────────────────────────┘
              ┌──────────────┴───────────────┐
              │ Yes                          │ No
              ▼                              ▼
    ┌──────────────────┐         ┌──────────────────────────┐
    │ Query expansion  │         │ Is the doc style         │
    │ (capsule 03)     │         │ predictable and domain-  │
    └──────────────────┘         │ specific?                │
                                 └────────────┬─────────────┘
                                              │
                              ┌───────────────┴────────────┐
                              │ Yes                        │ No
                              ▼                            ▼
                    ┌─────────────────┐         ┌──────────────────┐
                    │ HyDE 🎯         │         │ Query rewriting   │
                    │                 │         │ (capsule 04)      │
                    └─────────────────┘         └──────────────────┘

HyDE wins especially when:

• The domain has a predictable documentation style (technical docs, papers, manuals).
• The queries are open-ended and expect long answers.
• Recall is the metric you want to improve (not precision).
• Latency >700ms is acceptable.

HyDE does NOT win when:

• Simple factual queries ("what is X?") — the hypothetical doc would be trivial and doesn't help.
• Exact match matters (codes, IDs) — HyDE doesn't preserve exactness.
• Latency <500ms is required.
• Unstructured narrative domain (literature, informal conversation).

Traps and common mistakes

Trap 1: hypothetical docs that hallucinate

The mistake: the LLM generates a hypothetical doc with wrong information. E.g. for "what's the current version of FastAPI?", it makes up "FastAPI 5.0".

Symptom: HyDE searches with the embedding of a doc that says "FastAPI 5.0", when the corpus has "FastAPI 0.110". Poor match.

How to prevent it: HyDE is NOT for factual queries with concrete answers. It's for open-ended queries where the doc's "style" matters more than the correctness of the hypothetical answer. For factual queries, use direct search.

Trap 2: temperature too high = a creative but off-topic hypothetical doc

The mistake: temperature=1.0 for "diversity".

Symptom: the LLM generates a creative doc that drifts away from the query's topic. HyDE then searches for docs about the drifted topic.

How to prevent it: temperature=0.2-0.4. The hypothetical doc has to be consistent with the query, not creative.

Trap 3: hypothetical doc too short

The mistake: max_tokens=100. The hypothetical doc is 50 words long.

Symptom: the short doc's embedding lives closer to the queries region than to the region of real docs. HyDE loses its geometric advantage.

How to prevent it: a hypothetical doc of 200-400 words. Enough to imitate the "style" of a real doc.

Trap 4: using HyDE for trivial factual queries

The mistake: search_with_hyde("what is Python?").

Symptom: generating a hypothetical doc + 1 extra search adds 700ms and cost, for a query that direct search resolves perfectly.

How to prevent it: detect trivial factual queries and skip HyDE.

def needs_hyde(query: str) -> bool:
    word_count = len(query.split())
    is_definitional = any(query.lower().startswith(w) for w in ["what is", "qué es", "define"])
    
    if is_definitional and word_count <= 5:
        return False  # trivial factual query, no HyDE
    if word_count > 8 and word_count <= 25:
        return True  # medium-sized open-ended query, HyDE helps
    return False

Trap 5: measuring HyDE with recall alone

The mistake: you turn HyDE on, recall goes up 20%, you ship. You never looked at precision.

Symptom: HyDE finds more relevant docs (recall+) but it also brings in some tangentially related docs that weren't relevant (precision-). The system looks better but the downstream LLM gets confused.

How to prevent it: measure precision AND recall together. If precision drops >3 points, the recall gain isn't worth it without a re-ranker to filter.

Trap 6: HyDE with no rerank produces noisy context

The mistake: HyDE → top-5 → straight to the LLM.

Symptom: HyDE's top-5 can include docs that match the hypothetical doc's "style" but don't answer the user's exact query. The LLM sees partially relevant context.

How to prevent it: HyDE → top-20 → cross-encoder rerank with the original query → top-5. The rerank with the original query (not the hypothetical doc) favors real relevance.

def hyde_with_rerank(query: str, top_k: int = 5):
    # Broad HyDE retrieval
    hyde_doc = generate_hypothetical_document(query)
    results = collection.query(query_texts=[hyde_doc], n_results=20)
    candidates = results['documents'][0]

    # Rerank with the ORIGINAL QUERY (not the hyde_doc)
    reranked = cross_encoder_rerank(query, candidates, top_k=top_k)
    return reranked

Applied exercise

Scenario: you're the AI Engineer at an academic papers platform. The data:

• 500K computer science papers, indexed with OpenAI text-embedding-3-large
• Typical queries: researchers looking for work related to their projects
• Example queries:
  • "transformer architectures for time-series forecasting" (complex, open-ended)
  • "BERT vs GPT differences" (comparative)
  • "what is attention mechanism" (factual)
  • "recent advances in retrieval augmented generation 2024" (temporal + open-ended)

Metrics:

• Precision@10: 78%
• Recall@10: 64% (the problem — researchers complain they can't find relevant papers)

Your job:

1. Decide whether HyDE applies here. For which query types and for which ones not.
2. Design a pipeline with a dynamic skip.
3. Estimate impact and cost.

def smart_paper_search(query: str, top_k: int = 10) -> list[dict]:
    """Pipeline with the optimal technique per query type."""

    # Detect the type
    word_count = len(query.split())
    is_factual = any(query.lower().startswith(w) for w in ["what is", "define"])
    is_comparative = "vs" in query.lower() or "compare" in query.lower() or "difference" in query.lower()
    is_open_research = word_count > 5 and not is_factual and not is_comparative

    if is_factual and word_count <= 5:
        # Factual query: direct search
        results = collection.query(query_texts=[query], n_results=20)
        return cross_encoder_rerank(query, results['documents'][0], top_k=top_k)

    elif is_comparative:
        # Comparative query: decomposition
        return decompose_then_rerank(query, final_top_k=top_k)

    elif is_open_research:
        # Open-ended research query: HyDE + rerank
        hyde_doc = generate_hypothetical_document(query)
        results = collection.query(query_texts=[hyde_doc], n_results=30)
        # Rerank with the ORIGINAL query, not with hyde_doc
        return cross_encoder_rerank(query, results['documents'][0], top_k=top_k)

    else:
        # Default: direct search with rerank
        results = collection.query(query_texts=[query], n_results=20)
        return cross_encoder_rerank(query, results['documents'][0], top_k=top_k)

Category              %       Current recall   Recall with technique    Weighted gain
─────────────────────────────────────────────────────────────────────────────────────────
Open research        50%       60%              ~80% (HyDE+rerank)       +10 pts
Comparative          15%       55%              ~80% (decomposition)     +3.75 pts
Factual              25%       70%              ~75% (simple rerank)     +1.25 pts
Others               10%       65%              ~70%                     +0.5 pts

Total expected gain: 64% → ~80% recall@10 (+16 points)

Category         No optimization     With technique  Extra latency
──────────────────────────────────────────────────────────────────
Open research    400ms              1100ms          +700ms
Comparative      400ms              1300ms          +900ms
Factual          400ms              500ms           +100ms
Others           400ms              500ms           +100ms

Weighted average latency: ~860ms (vs 400ms baseline)

queries_per_day = 2000  # estimate
days_per_month = 30

# Distribution
PCT_HYDE = 0.50  # open research → HyDE
PCT_DECOMP = 0.15  # comparative → decomposition
PCT_DIRECT = 0.35  # factual + others → direct

# Costs per LLM call
COST_HYDE_LLM = 0.0005      # gpt-4o-mini, ~500 tokens out
COST_DECOMP_LLM = 0.0003    # decomposition decision
COST_RERANK = 0             # local cross-encoder, free

monthly_cost = (
    queries_per_day * days_per_month * PCT_HYDE * COST_HYDE_LLM +
    queries_per_day * days_per_month * PCT_DECOMP * COST_DECOMP_LLM
)
print(f"Monthly cost: ${monthly_cost:.2f}")

Recap and next step

What you learned:

• HyDE searches with the embedding of an LLM-generated hypothetical document, instead of the query.
• Geometric insight: doc-to-doc cosine ~0.88 vs query-to-doc cosine ~0.72. A better match.
• Typical gain: +15-25% recall, especially in domains with a predictable doc style.
• Cost: ~700ms latency + ~$0.001 per query (the LLM call for generation).
• Variants: Multi-HyDE (3 docs in parallel + RRF) and Hybrid (query + HyDE fused).
• HyDE is NOT useful for trivial factual queries or for exact matching of identifiers.
• Optimal combination: HyDE for broad retrieval + cross-encoder rerank with the original query (not with hyde_doc).
• Main trap: hypothetical docs that hallucinate on factual queries. A dynamic skip prevents it.

Checkpoint: before moving on, you should be able to:

• Explain why searching with a hypothetical doc gets a better cosine than searching with the direct query.
• Implement HyDE with structured outputs and a dynamic skip.
• Decide when HyDE beats query expansion, rewriting or decomposition.

Next capsule: 07 — Comparing query optimization techniques.

We covered the four main techniques: expansion, rewriting, decomposition, HyDE. Capsule 07 puts them side by side with a reproducible decision framework. It's the capsule you'll come back to when picking the right technique for a given scenario.

Resources

1. HyDE Paper — Precise Zero-Shot Dense Retrieval (Gao et al., 2022) — The original paper
2. Pinecone — HyDE Explained — Visual tutorial with benchmarks
3. LangChain — HyDE Implementation — Reference implementation
4. LlamaIndex — HypotheticalDocumentEmbedder — The pattern in LlamaIndex
5. Anthropic — Contextual Retrieval — Complementary technique
6. Stanford NLP — Dense Retrieval Surveys — Foundational on query-document matching

Estimated time: 30-35 minutes Next: 07-technique-comparison-1.md