Capsule 07: Decision framework — which query optimization technique to pick

Capsule overview

We covered the four main query optimization techniques in this module: expansion (capsule 03), rewriting (04), decomposition (05), HyDE (06). Each with its trade-offs, use cases, failure modes. This capsule is the operational consolidation: a decision framework to pick the right technique for a given scenario, benchmarks side by side, and combination patterns for when a single technique isn't enough.

It's the capsule you'll come back to every time you diagnose that your RAG system needs query optimization. It saves you from rereading the previous 4 capsules to decide.

By the end of this capsule you'll be able to:

• ✅ Compare the four techniques across six dimensions (recall, precision, latency, cost, complexity, domain)
• ✅ Apply a decision flowchart to pick a technique in under 10 minutes
• ✅ Design hybrid pipelines that chain two or more techniques
• ✅ Compute the monthly TCO of each option for a given volume
• ✅ Identify the three most common anti-patterns when combining techniques
• ✅ Tell apart when the problem needs query optimization vs when it's a retrieval or re-ranking problem

Estimated time: 25-30 minutes

Consolidated benchmark of the 4 techniques

Quality

Cost and latency

Operational complexity

Decision framework

                          What's the dominant problem?
                                     │
        ┌────────────────────────────┼────────────────────────────┐
        │                            │                            │
        ▼                            ▼                            ▼
   Low RECALL                 Low PRECISION               COMPLEX QUERIES
        │                            │                            │
        ▼                            ▼                            ▼
┌────────────────┐         ┌────────────────┐         ┌────────────────────┐
│ Short and       │         │ Keyword-style  │         │ Multi-aspect,      │
│ ambiguous       │         │ or incomplete  │         │ comparative or     │
│ queries?        │         │ queries?       │         │ multi-step?        │
└───────┬────────┘         └───────┬────────┘         └─────────┬──────────┘
        │                          │                            │
   ┌────┴────┐                  ┌──┴──┐                  ┌──────┴──────┐
   │ YES     │ NO                │ YES │ NO                │ YES         │ NO
   ▼         ▼                  ▼     ▼                  ▼              ▼
┌──────┐  ┌──────┐          ┌──────┐ │            ┌──────────────┐  │
│Expan-│  │HyDE  │          │Rewri-│ │            │Decomposition │  │
│sion  │  │      │          │ting  │ │            │              │  │
└──────┘  └──────┘          └──────┘ │            └──────────────┘  │
                                     │                                │
                                     ▼                                ▼
                          ┌──────────────────────┐         ┌──────────────────┐
                          │ Latency <500ms?      │         │ Consider:        │
                          │  └─ Yes: skip optim. │         │ - Re-ranking      │
                          │  └─ No: HyDE         │         │ - Hybrid search   │
                          └──────────────────────┘         │ - Metadata filter │
                                                           └──────────────────┘

Applied to real cases

Hybrid patterns

Sometimes a single technique doesn't solve the problem. Three proven patterns:

Pattern 1: Rewriting → Expansion (cascade)

For queries that are badly phrased AND ambiguous (e.g. keyword-style + short).

def rewrite_then_expand(query: str) -> list[str]:
    """Rewrite first, then expand the rewrite."""
    # Step 1: convert to natural language
    rewritten = rewrite_structural(query).rewritten

    # Step 2: if it's still ambiguous, expand
    if len(rewritten.split()) <= 6:
        expansion = expand_query(rewritten, num_expansions=4)
        return [rewritten] + expansion.queries

    return [rewritten]

Use case: "k8s pod fail" (keyword + ambiguous):

1. Rewriting → "Why is my Kubernetes pod failing?"
2. Expansion → 4 queries covering different failure reasons

Pattern 2: HyDE + Decomposition

For queries that are open-ended AND multi-aspect (e.g. complex research questions).

def hyde_with_decomposition(query: str, top_k: int = 8):
    """If the query is decomposable, decompose first, then HyDE for each sub."""
    decomp = decompose_query(query)

    if not decomp.is_decomposable:
        # Simple query → HyDE directly
        return hyde_search(query, top_k)

    # Complex query → HyDE for each sub-query
    all_results = []
    for sub_q in decomp.sub_queries:
        hyde_doc = generate_hypothetical_document(sub_q)
        results = collection.query(query_texts=[hyde_doc], n_results=10)
        all_results.append(results['ids'][0])

    # Fuse with RRF
    fused = reciprocal_rank_fusion(all_results)
    top_ids = [doc_id for doc_id, _ in fused[:top_k]]
    return collection.get(ids=top_ids)

Use case: "Compare transformer architectures and recommend the best for time-series forecasting":

1. Decomposition → 3 sub-queries
2. HyDE for each sub-query → find specific papers
3. Fusion → final top-K

Pattern 3: Adaptive (the LLM classifies and picks)

The LLM decides which technique to apply per query:

def adaptive_query_optimization(query: str, chat_history: list = None):
    """The LLM classifies the query and applies the optimal technique."""

    # Classification with an LLM
    classification = classify_query_type(query, chat_history)

    if classification == "well_formed":
        return query  # skip optimization

    elif classification == "ambiguous":
        expansion = expand_query(query, num_expansions=4)
        return [query] + expansion.queries

    elif classification == "incomplete":
        if chat_history:
            return [rewrite_with_context(query, chat_history).rewritten]
        else:
            return [rewrite_structural(query).rewritten]

    elif classification == "complex":
        decomp = decompose_query(query)
        if decomp.is_decomposable:
            return decomp.sub_queries
        return [query]

    elif classification == "open_research":
        # HyDE is a good fit for research questions
        return ["use_hyde"]  # signal to use HyDE instead of the direct query

    else:
        return [query]

Trade-off: more cost (an extra LLM call for classification) but better selection. Useful when your traffic has a wide mix of query types.

Computing the monthly TCO

To decide between options, compute the concrete monthly cost:

# tco_query_optimization.py
QUERIES_PER_DAY = 10_000
DAYS_PER_MONTH = 30

# Costs per query (gpt-4o-mini, May 2026)
COST_BY_TECHNIQUE = {
    "no_optimization": 0,
    "rewriting": 0.0007,
    "expansion": 0.0011,
    "decomposition": 0.0020,
    "hyde": 0.0015,
    "adaptive": 0.0010,  # average depending on the mix
}

# Extra latency
LATENCY_BY_TECHNIQUE = {
    "no_optimization": 0,
    "rewriting": 500,
    "expansion": 750,
    "decomposition": 1100,
    "hyde": 800,
    "adaptive": 700,
}

# Typical recall gain
RECALL_IMPROVEMENT_BY_TECHNIQUE = {
    "no_optimization": 0,
    "rewriting": 4,
    "expansion": 17,
    "decomposition": 12,
    "hyde": 22,
    "adaptive": 18,
}


def calculate_tco_and_impact(technique: str, baseline_recall: float = 0.65):
    monthly_queries = QUERIES_PER_DAY * DAYS_PER_MONTH

    cost_monthly = monthly_queries * COST_BY_TECHNIQUE[technique]
    new_recall = baseline_recall + RECALL_IMPROVEMENT_BY_TECHNIQUE[technique] / 100
    latency = LATENCY_BY_TECHNIQUE[technique]

    return {
        "technique": technique,
        "monthly_cost_usd": round(cost_monthly, 2),
        "expected_recall": f"{new_recall:.0%}",
        "extra_latency_ms": latency,
    }


for tech in COST_BY_TECHNIQUE.keys():
    result = calculate_tco_and_impact(tech)
    print(f"\n{tech.upper()}:")
    for k, v in result.items():
        print(f"  {k}: {v}")

Output (10K queries/day):

NO_OPTIMIZATION:
  monthly_cost_usd: 0.00
  expected_recall: 65%
  extra_latency_ms: 0

REWRITING:
  monthly_cost_usd: 210.00
  expected_recall: 69%
  extra_latency_ms: 500

EXPANSION:
  monthly_cost_usd: 330.00
  expected_recall: 82%
  extra_latency_ms: 750

DECOMPOSITION:
  monthly_cost_usd: 600.00
  expected_recall: 77%
  extra_latency_ms: 1100

HYDE:
  monthly_cost_usd: 450.00
  expected_recall: 87%
  extra_latency_ms: 800

ADAPTIVE:
  monthly_cost_usd: 300.00
  expected_recall: 83%
  extra_latency_ms: 700

How to read it:

• HyDE gives the best recall (+22pts) for $450/month — a good option if recall is critical.
• Expansion gives +17pts for $330/month — the price/quality sweet spot.
• Decomposition is expensive and only justified if you have a lot of complex queries.
• Adaptive gives almost the same result as HyDE/Expansion mixed, at a lower cost — but it takes more operational complexity.

Anti-patterns when combining techniques

Anti-pattern 1: applying every technique

The mistake: "let's add rewriting + expansion + HyDE for maximum quality".

Symptom: +2 seconds of latency, 4x the cost, a marginal gain over the best individual technique.

Why it happens: the techniques attack different problems. If your dominant problem is recall (HyDE solves it), adding rewriting only adds latency with no benefit.

How to prevent it: diagnose first (capsule 02), apply the technique that attacks the dominant problem. Combine only when there's evidence of multiple problems.

Anti-pattern 2: combining without measuring

The mistake: you assume two combined techniques add up their gains. Rewriting +5% + Expansion +15% = +20% combined.

Reality: the techniques overlap. Combined, they typically give +18% (not +20%) because of the interaction. Sometimes combining them makes things WORSE: one technique "breaks" what the other one fixes.

How to prevent it: A/B test each combination against each individual technique on an eval set. If the combination doesn't beat the best individual one by >5%, the extra complexity isn't worth it.

Anti-pattern 3: optimization where the query isn't the problem

The mistake: low precision/recall → you assume it's a query problem → you add 4 optimization techniques.

Reality: many times the problem is:

• Bad chunking (M02) — chunks too big/too small
• Weak embeddings (M07) — a model misaligned with the domain
• Missing re-ranking (M04) — the top-5 has false positives
• Missing hybrid search (M05) — queries with exact keywords
• An incomplete corpus — the answer isn't in the documents

How to prevent it: diagnose where the problem is before investing in query optimization. If the problem is re-ranking, adding HyDE won't help.

def diagnose_pipeline_bottleneck(eval_set):
    """Identify which pipeline component is the bottleneck."""
    issues = {"missing_data": 0, "retrieval": 0, "reranking": 0, "query": 0}

    for item in eval_set:
        # Does the right info exist in any chunk of the corpus?
        in_corpus = check_corpus(item["expected_text"])
        if not in_corpus:
            issues["missing_data"] += 1
            continue

        # Does cosine find the right doc in the top-50?
        results = collection.query(query_texts=[item["query"]], n_results=50)
        if not any(item["expected_id"] in id_ for id_ in results['ids'][0]):
            issues["retrieval"] += 1
            continue

        # Does the re-rank find the right doc in the top-5?
        reranked = cross_encoder_rerank(item["query"], results['documents'][0], top_k=5)
        if not any(item["expected_id"] in r.original_index for r in reranked):
            issues["reranking"] += 1
            continue

        # If we got this far, the doc is in the top-5 with rerank — the query problem was the smaller one
        issues["query"] += 1

    print("Bottleneck diagnosis:")
    for issue, count in issues.items():
        print(f"  {issue}: {count}/{len(eval_set)}")

If "retrieval" is high, query optimization helps. If "reranking" is high, add/improve the reranker. If "missing_data" is high, expand the corpus — no query technique solves that.

Traps and common mistakes

Trap 1: applying query optimization with no prior diagnosis

Covered above. Diagnose first.

Trap 2: copying a technique from a blog without measuring

The mistake: you see that "HyDE gives +25% recall" in a blog post, and you apply it. Your real gain is +5%.

How to prevent it: blog benchmarks are on specific datasets/domains. Measure on your eval set before committing.

Trap 3: optimization that breaks simple queries

The mistake: you turn expansion on for every query. Simple queries like "Stripe API documentation" get expanded into 5 versions, each vaguer than the original. Recall goes up on complex queries but down on simple ones.

How to prevent it: a dynamic skip (seen in capsules 03, 04, 05, 06). Well-formed queries don't need optimization.

Trap 4: caching the optimization without invalidating when the prompt changes

The mistake: you cache query expansion results. You change the LLM's prompt. The cached queries keep using expansions from the old prompt.

How to prevent it: version the prompt in the cache key:

PROMPT_VERSION = "v3"

def cache_key(query: str) -> str:
    return f"{PROMPT_VERSION}:{hashlib.md5(query.encode()).hexdigest()}"

Trap 5: measuring recall alone, without precision

The mistake: you turn HyDE on, recall goes up 20%, you ship.

Symptom: precision drops 5% (HyDE found tangential docs). The downstream LLM gets confused by the extra context. Quality gets worse.

How to prevent it: measure precision AND recall together. If the combination doesn't improve both, something's wrong.

Trap 6: optimization with no re-ranking

The mistake: you turn expansion + HyDE on but you have no re-ranking. The fused results go straight to the LLM.

Symptom: the LLM gets 10 moderately relevant docs instead of 5 highly relevant ones. Answer quality is worse than with no optimization.

How to prevent it: query optimization and re-ranking work together. Optimization improves retrieval, re-ranking refines the candidates. Without re-ranking, optimization can degrade quality.

Applied exercise

Scenario: you're the AI Engineer at a corporate education platform. The RAG system answers employee questions about technology courses.

The data:

• 50K chunked courses
• 20K queries/day
• Current pipeline: cosine + cross-encoder rerank

Log analysis:

Category                        %      Examples
─────────────────────────────────────────────────────────────
Well-formed                    30%      "what does useState do in React?"
Keyword-style                  25%      "react useState hook"
Incomplete (chat)              20%      "how do I configure it?" in a Docker chat
Comparative                    12%      "differences between useState and useReducer"
Vague                           8%      "help with my code"
Multi-step                      5%      "how do I test and deploy my app"

Metrics:

• Precision@5: 79%
• Recall@5: 61%
• Stakeholders want recall@5 ≥ 80% without going over 1500ms p95.

Your job:

1. Apply the decision framework. What pipeline do you design?
2. Compute the expected cost and latency.
3. A validation plan.

def adaptive_pipeline(query: str, chat_history: list = None, top_k: int = 5):
    """
    Adaptive pipeline: the LLM classifies and applies the optimal technique.
    """
    # Classification with an LLM (gpt-4o-mini, ~150ms)
    category = classify_query(query, chat_history)

    queries_to_search = []

    if category == "well_formed":
        # Skip optimization
        queries_to_search = [query]

    elif category == "keyword_style":
        # Structural rewriting
        rewritten = rewrite_structural(query).rewritten
        queries_to_search = [rewritten]

    elif category == "incomplete" and chat_history:
        # Rewriting with context
        rewritten = rewrite_with_context(query, chat_history).rewritten
        queries_to_search = [rewritten]

    elif category == "comparative":
        # Decomposition
        decomp = decompose_query(query)
        queries_to_search = decomp.sub_queries if decomp.is_decomposable else [query]

    elif category == "vague":
        # Expansion (vague queries → several interpretations)
        expansion = expand_query(query, num_expansions=4)
        queries_to_search = [query] + expansion.queries

    elif category == "multi_step":
        # Decomposition
        decomp = decompose_query(query)
        queries_to_search = decomp.sub_queries if decomp.is_decomposable else [query]

    # Retrieval for all the queries
    if len(queries_to_search) == 1:
        results = collection.query(query_texts=queries_to_search, n_results=20)
        candidates = results['documents'][0]
    else:
        # Multiple queries → parallel + RRF
        all_rankings = parallel_retrieve(queries_to_search, top_k=15)
        fused = reciprocal_rank_fusion(all_rankings)
        top_ids = [doc_id for doc_id, _ in fused[:30]]
        candidates = collection.get(ids=top_ids)['documents']

    # Re-rank with the original query
    return cross_encoder_rerank(query, candidates, top_k=top_k)

queries_per_day = 20_000
days_per_month = 30
total_queries = queries_per_day * days_per_month  # 600K

# Classification cost (all the queries)
classification_cost_per_query = 0.00005  # gpt-4o-mini, short prompt
classification_total = total_queries * classification_cost_per_query

# Cost of each technique
costs_by_technique = {
    "skip": 0,
    "rewrite": 0.0006,
    "rewrite_ctx": 0.0008,
    "decomp": 0.0018,
    "expansion": 0.0011,
}

distribution = {
    "skip": 0.30,
    "rewrite": 0.25,
    "rewrite_ctx": 0.20,
    "decomp": 0.17,  # comparative + multi-step
    "expansion": 0.08,
}

technique_total = sum(
    total_queries * pct * costs_by_technique[tech]
    for tech, pct in distribution.items()
)

monthly_cost = classification_total + technique_total
print(f"Classification cost: ${classification_total:.2f}")
print(f"Technique cost: ${technique_total:.2f}")
print(f"Total monthly: ${monthly_cost:.2f}")

Classification cost: $30.00
Technique cost: $440.40
Total monthly: $470.40

Category              %     Current recall   Recall with technique
──────────────────────────────────────────────────────────────────
Well-formed         30%      75%              75% (unchanged)
Keyword             25%      55%              78% (rewriting)
Incomplete          20%      45%              82% (rewriting ctx)
Comparative         12%      58%              80% (decomposition)
Vague                8%      40%              72% (expansion)
Multi-step           5%      55%              78% (decomposition)

Expected global recall:
0.30(0.75) + 0.25(0.78) + 0.20(0.82) + 0.12(0.80) + 0.08(0.72) + 0.05(0.78)
= 0.225 + 0.195 + 0.164 + 0.096 + 0.058 + 0.039
= 0.777 (78%) → close to the 80% target

Recap and next step

What you learned:

• Four query optimization techniques, each for a different problem: expansion (ambiguity), rewriting (incomplete/keyword), decomposition (complex), HyDE (maximum recall).
• A decision framework based on the dominant problem (recall vs precision vs complex queries).
• Hybrid patterns for multiple problems: cascade (rewriting + expansion), parallel (HyDE per sub-query).
• Adaptive with an LLM classifier is the answer when the traffic is very varied.
• Typical monthly TCO for 10K queries/day: $200-600 depending on the technique. Negligible for most products.
• Anti-patterns: applying every technique, combining without measuring, optimizing where the query isn't the problem.
• Diagnose the pipeline's bottleneck before investing in query optimization.

Checkpoint: before closing the module, you should be able to:

• Apply the decision framework to pick a technique in under 10 minutes.
• Design hybrid pipelines that combine two or more techniques with justification.
• Compute the monthly TCO of query optimization for your volume.
• Diagnose whether low precision/recall is a query problem vs retrieval vs rerank.

Next capsule: 08 — Query Optimization Pipeline project.

The module's closer: you'll build an end-to-end query optimization pipeline with A/B testing between two techniques, measurement on your own eval set, and a comparative report. It's the tool you'll apply on day 1 of any RAG project where you suspect the user's queries are the bottleneck.

Resources

1. LangChain — Query Construction Guide — Advanced patterns
2. LlamaIndex — Query Pipelines — Reference implementations
3. Pinecone — Query Optimization Series — Full tutorial
4. Anthropic — Contextual Retrieval — Complementary technique
5. Goodhart's Law — Why optimizing what you measure can break what you don't
6. Microsoft — Query Optimization in RAG — A use case with Azure

Estimated time: 25-30 minutes Next: 08-project-query-optimizer.md