Capsule 02: Why the user's query is almost never the optimal query

Capsule overview

There's a myth in RAG: that the user's query is sacred input and your only job is to "search better". The reality is exactly the opposite. The query the user types is the pipeline's first point of failure, not its last. If you don't touch it before embedding and searching, you're accepting a quality ceiling that no re-ranker, no hybrid search, no chunking can ever climb back over.

This capsule shows you the three patterns by which direct queries degrade retrieval — and why understanding this is the prerequisite for appreciating the techniques that follow (query expansion, rewriting, decomposition, HyDE). Without this "why", those techniques sound like unnecessary complexity. With this "why", they become obviously necessary.

By the end of this capsule you'll be able to:

• ✅ Identify the three patterns of problematic queries: ambiguous, incomplete, badly phrased
• ✅ Diagnose why a specific user query is producing bad results
• ✅ Quantify the impact of each problem on recall and precision
• ✅ Anticipate which query optimization technique to apply to each type of problem
• ✅ Distinguish a query problem from a retrieval problem (they aren't always the same thing)
• ✅ Spot the mistake of "the user should write better queries" — you're responsible, not the user

Estimated time: 25-30 minutes

The insight: the user's query is optimized for Google, not for your RAG

Users learned to google 25 years ago. They type short, keyword-stuffed queries, optimized for traditional search engines. Your RAG operates under different principles: semantic embeddings that prefer complete sentences in natural language. What the user does by instinct is exactly the opposite of what your system needs.

What the user types:            "fastapi auth"          (3 tokens, ambiguous)
What your RAG would want:       "How do I implement OAuth2 authentication with
                                 OAuth2PasswordBearer in FastAPI?"   (a complete, specific sentence)

Recall with the first:          52%
Recall with the second:         87%
The difference:                 35 points

You can't teach millions of users to type "optimal" queries. What you can do is transform the user's query before you embed it. That's query optimization. But before you learn the techniques, you have to understand exactly which problems they solve.

Problem 1: ambiguous queries (multiple interpretations)

The user's query: "fastapi auth"

What did they mean?

• "How to implement OAuth2 authentication in FastAPI"
• "FastAPI with JWT tokens"
• "FastAPI with API keys"
• "FastAPI with basic auth"
• "FastAPI with session cookies"
• "How to test authentication in FastAPI"
• "A comparison of auth methods in FastAPI"

That's 7 different interpretations, each with different answers in your corpus. Cosine similarity has no idea which one the user wants — it embeds the 2 tokens and returns a mix of docs across all 7 topics, none of them in any depth.

Why it happens

Embeddings encode the semantic centroid of the tokens. "fastapi auth" → a vector pointing at the "general region of FastAPI + authentication". The docs specifically about OAuth2 sit in one sub-region. The JWT ones in another. The basic auth ones in another. The generic query is equidistant from all three — so it returns a bit of each.

The measurable impact

Query: "fastapi auth"
─────────────────────────────────────────────
Top 5 results:
  1. "OAuth2 in FastAPI overview"      (relevant to 1 interpretation)
  2. "JWT tokens with FastAPI"         (relevant to 1 interpretation)
  3. "API keys best practices"         (relevant to 1 interpretation)
  4. "FastAPI security overview"       (generic)
  5. "Basic auth tutorial"             (relevant to 1 interpretation)

If the user specifically wanted OAuth2:
  - Doc 1 is OK (an overview, no depth)
  - Docs 2-5 are noise for them
  - Effective recall: 1/5 = 20%

The general pattern: queries of 1-3 tokens are ~40% more likely to be ambiguous than queries of 7+ tokens.

The technique that fixes it

Query expansion (capsule 03): generate several expanded versions of the original query, search with each one, and fuse the results. If the user typed "fastapi auth", expand it to:

• "FastAPI OAuth2 authentication"
• "FastAPI JWT authentication"
• "FastAPI API key authentication"

And combine the rankings. It covers the likely interpretations without requiring the user to be explicit.

Problem 2: incomplete queries (missing critical context)

The user's query: "how to deploy"

Deploy what? Where? With which tools? In which environment? The user assumes the context is implicit ("I'm chatting with a FastAPI bot, obviously I'm asking about deploying FastAPI"). The system has none of that context.

Why it happens

Incomplete queries come from:

1. Implicit prior conversation: the user assumes the system "remembers" the topic of the previous messages.
2. Product context: a Stripe user doesn't write "Stripe payment integration", they write "how to charge", assuming the product context is obvious.
3. A cognitive shortcut: the user knows what they want and omits the terms that are "obvious" to them.

The measurable impact

Query: "how to deploy"
─────────────────────────────────────────────
With no context, the embedding points at "deployment in general".

Top 5 results:
  1. "AWS Lambda deployment guide"     (deployment, but is it what they want?)
  2. "Docker deployment basics"        (deployment, but is it what they want?)
  3. "Kubernetes for Python apps"      (deployment, but is it what they want?)
  4. "CI/CD setup with GitHub Actions" (deployment, but is it what they want?)
  5. "Heroku deployment for beginners" (deployment, but is it what they want?)

If the user is in a FastAPI + Docker context:
  - Only doc 2 is relevant
  - Effective precision: 1/5 = 20%

The general pattern: queries of 2-4 tokens with no proper nouns or specific technologies are ~50% more likely to be incomplete.

The technique that fixes it

Query rewriting with context (capsule 04): an LLM takes the user's query + whatever context is available (chat history, product identity, the last topic) and rewrites it into a complete query.

# Without rewriting
user_query = "how to deploy"
results = retrieve(user_query)  # Recall: 35%

# With rewriting (the LLM has the chat's context)
chat_history = ["I'm building a FastAPI app", "I want to use Docker"]
rewritten = llm_rewrite(user_query, chat_history)
# rewritten = "How to deploy a FastAPI application with Docker"
results = retrieve(rewritten)  # Recall: 87%

Problem 3: badly phrased queries (keyword style vs natural language)

The user's query: "fastapi async performance"

That's keyword style — users trained on Google strip out the stop words ("how", "is", "a", "the") because they know Google ignores them. But embeddings work the other way around: those "unnecessary" words are exactly what help the model understand the question's structure.

Why it happens

# Keyword style
"fastapi async performance"
# The model embeds these 3 tokens. It captures: "topic = FastAPI, async, performance"
# It does NOT capture: "this is a question about how these concepts relate"

# Natural language
"How does FastAPI achieve high performance through async/await?"
# The model embeds the sentence. It captures: "topic = FastAPI async performance"
# + "intent = an explanation of a causal mechanism"
# + "structure = a process question ('how does X achieve Y')"

Modern embedding models (OpenAI text-embedding-3-small, Cohere, etc.) were trained on natural text, not keyword lists. They do far better with complete sentences.

The measurable impact

Keyword query:     "fastapi async performance"     →  Recall: 48%
Natural query:     "How does FastAPI achieve high performance with async/await?"
                                                   →  Recall: 72%
The difference:    +24 points

The technique that fixes it

Structural query rewriting (capsule 04): convert keyword queries into natural interrogative sentences before embedding them.

For highly technical queries that need even more context: HyDE — Hypothetical Document Embeddings (capsule 06). Instead of embedding the query, you generate a hypothetical answer with an LLM and embed that answer. What you're searching the corpus for is "documents similar to the kind of answer an expert would give to this" — which is semantically much closer to the real docs.

The quantitative comparison

The key reading: ~70-80% of real queries have at least one of these problems. Going from ignoring them to remedying them transforms the system's quality, typically +25-35 points of recall.

The expensive mistake: blaming the user

When stakeholders see low precision, the first reaction is sometimes:

"Users don't know how to type specific queries. We need to educate them."

This is a trap for three reasons:

1. It doesn't scale. You have thousands or millions of users. You aren't going to train them.
2. It isn't the user's problem. Users are fine typing the way they type. The system has to adapt to them, not the other way around.
3. It hides the real problem. While you're busy blaming the user, your pipeline stays suboptimal. When a competitor implements query optimization, they eat your market.

The correct framing: the user's query is input. Optimizing it before you process it is the pipeline's responsibility, exactly like validating inputs in a REST API. Nobody tells a REST endpoint's client "learn to write better JSON" — you sanitize and validate. Same principle here.

How to diagnose which problem hits your system hardest

Before you apply query optimization techniques blindly, measure it:

# diagnose_query_quality.py
from collections import Counter

def categorize_query(query: str, llm_classifier=None) -> str:
    """
    Classifies a query into one of the problematic patterns.
    For accuracy, use an LLM (gpt-4o-mini) as the classifier.
    """
    # Simple heuristics
    word_count = len(query.split())

    if word_count < 4:
        # Probably ambiguous or incomplete
        if any(name in query.lower() for name in PRODUCT_NAMES):
            return "ambiguous"  # it has the product but lacks detail
        return "incomplete"

    # Detecting keyword style: the ratio of keywords to stop words
    stop_words_count = sum(1 for w in query.lower().split() if w in {"how", "is", "the", "a", "what", "when", "where"})
    if stop_words_count == 0 and word_count <= 6:
        return "keyword_style"

    return "well_formed"


def diagnose_query_problems(query_log: list[str]) -> dict:
    """
    Over a sample of real queries, what proportion has each problem?
    """
    counts = Counter()
    for q in query_log:
        category = categorize_query(q)
        counts[category] += 1

    total = len(query_log)
    return {
        cat: f"{count} ({count/total:.0%})"
        for cat, count in counts.items()
    }


# Over 1000 real queries from the production log
diagnosis = diagnose_query_problems(production_queries[:1000])
print(diagnosis)

Typical output:

{
    "incomplete": "412 (41%)",
    "keyword_style": "287 (29%)",
    "ambiguous": "208 (21%)",
    "well_formed": "93 (9%)",
}

How to read it: 91% of the queries have problems. The most frequent are incomplete (41%) and keyword-style (29%). Your optimization priority should be query rewriting — it would attack 70% of the problematic queries.

Traps and common mistakes

Trap 1: applying query optimization with no diagnosis

The mistake: you copy a blog that says "implement HyDE" and bolt it onto the pipeline.

The symptom: HyDE adds 800ms of latency and an extra LLM cost, and the gain is marginal because your dominant problem was incomplete queries, not badly phrased ones.

How to prevent it: diagnose first (see the script above). Apply the technique that attacks the dominant problem.

Trap 2: treating query optimization as a panacea

The mistake: you assume that with query optimization you don't need re-ranking, hybrid search, or decent chunking.

The reality: they're complementary. Query optimization improves retrieval's input. Re-ranking refines its output. Chunking defines which units get searched. Hybrid search complements the embeddings with keywords.

How to prevent it: measure each component against your eval set. Query optimization is typically the highest-impact improvement if your base pipeline is semantic + cosine, but it rarely solves everything.

Trap 3: optimizing queries that were already fine

The mistake: you apply query rewriting to EVERY query, including the well-formed ones.

The symptom: queries that were already good go through LLM rewriting unnecessarily, adding 200-500ms and cost. Sometimes the LLM "rewrite" turns a good query into a worse one.

How to prevent it: classify first, then optimize only the problematic ones. Well-formed queries go straight to retrieval.

def smart_pipeline(query: str):
    category = categorize_query(query)
    
    if category == "well_formed":
        return retrieve(query)  # skip optimization
    elif category == "ambiguous":
        return retrieve_with_expansion(query)
    elif category == "incomplete":
        return retrieve_with_rewriting(query, context)
    elif category == "keyword_style":
        return retrieve_with_rewriting(query, style="natural")

Trap 4: rewriting that changes the user's intent

The mistake: an LLM "rewrites" "why FastAPI is slow" into "How to make FastAPI faster". It changed the question.

The symptom: the system answers "to make FastAPI faster, do X" when the user wanted to understand why it's sometimes slow.

How to prevent it: conservative prompts in query rewriting — "expand the query, do NOT change its intent". Plus human validation over the eval set.

Trap 5: sending queries in other languages to an English rewriter

The mistake: your rewriter uses GPT-4 with an English prompt. It receives a query in Spanish. The LLM "rewrites" it into English.

The symptom: the translated query doesn't match the Spanish docs well. Recall collapses on multilingual queries.

How to prevent it: use a prompt in the same language as the query, or an explicit instruction: "preserve the query language".

Trap 6: forgetting the aggregate cost

The mistake: you add query expansion (5 queries per original query) + LLM rewriting (1 LLM call) + HyDE (1 LLM call). Every user query now makes 7 downstream calls.

The symptom: costs multiply 7x. The user's time-to-first-hit rises from 200ms to 2.5 seconds.

How to prevent it: estimate the total cost before you combine techniques. Every technique has to justify itself independently against the eval set.

Applied exercise

The scenario: you're an AI Engineer at a company that provides technical support for DevOps tools. The data:

• A RAG system in production with cosine + cross-encoder rerank
• Current Precision@5: 78%
• Current Recall@5: 62%
• No query optimization
• 5,000 queries/day

The analysis of the query log (a 1000-query sample):

Category:         Count       Examples
─────────────────────────────────────────────────────────
incomplete        510 (51%)   "fix this error", "how to deploy", "auth not working"
keyword_style     320 (32%)   "kubernetes pod restart", "docker compose env"
ambiguous         140 (14%)   "ssl error", "timeout"
well_formed        30 (3%)    "How do I configure Helm chart values for staging?"

Your job:

1. Diagnose the dominant problem.
2. Propose 2-3 query optimization techniques in priority order, justified with the data.
3. Estimate the extra cost and the expected impact.

# The user is usually mid-conversation with the bot
# Use the recent history to rewrite the incomplete queries

REWRITER_PROMPT = """The user is in a DevOps support chat. Their previous messages:
{chat_history}

Their current query: "{user_query}"

If the query is incomplete (missing tools, technologies, or context), rewrite it
as a complete question. If it's already complete, return it unchanged.
Preserve the user's intent. Output only the rewritten query."""

def rewrite_query_with_context(user_query, chat_history):
    response = openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": REWRITER_PROMPT.format(
                chat_history="\n".join(chat_history[-3:]),
                user_query=user_query
            )},
        ],
        temperature=0.0
    )
    return response.choices[0].message.content.strip()

# Apply it only to the queries detected as "incomplete"
if categorize(query) == "incomplete":
    query = rewrite_query_with_context(query, chat_history)

KEYWORD_TO_NATURAL_PROMPT = """Convert this keyword-style query into a complete natural-language question.
Preserve all keywords. Do not add information that wasn't in the original.

Keyword query: {user_query}
Natural question:"""

def keyword_to_natural(user_query):
    if categorize(query) == "keyword_style":
        return llm_rewrite(user_query, KEYWORD_TO_NATURAL_PROMPT)
    return user_query

# Example:
# Input:  "kubernetes pod restart"
# Output: "How do I restart a Kubernetes pod?"

Summary and next step

What you learned:

• The user's query is an input that needs processing, not an untouchable sacred artifact.
• Three dominant patterns: ambiguous (multiple interpretations), incomplete (missing context), badly phrased (keyword vs natural).
• ~70-90% of real queries have at least one of these problems. It's worth diagnosing and attacking them.
• Every problem has a specific technique that fixes it: expansion (ambiguous), rewriting with context (incomplete), structural rewriting (badly phrased), HyDE (highly technical queries).
• Diagnose first, optimize second. Applying techniques blindly adds complexity with no guaranteed gain.
• Blaming the user for writing bad queries is a trap — the system is responsible for adapting to the user.
• Optimizing queries only when they're problematic (a dynamic skip) saves both cost and latency.

Checkpoint: before moving on, you should be able to:

• Classify a given query into one of the three problematic patterns.
• Predict which optimization technique would best attack the dominant problem in a query log.
• Diagnose whether low precision/recall comes from bad queries or bad retrieval.

Next capsule: 03 — Query Expansion.

You've just identified the problems. Capsule 03 covers the first fix: query expansion. Take an original query, generate several versions that cover the different likely interpretations, then fuse the results from each. It's the main weapon against ambiguous queries, and your first concrete experiment in query optimization.

Resources

1. Stanford NLP — Query Reformulation — Foundational reading on query expansion and reformulation
2. Anthropic — Contextual Retrieval — A related technique that improves retrieval by contextualizing chunks
3. Pinecone — Query Optimization Guide — A practical tutorial
4. LangChain — Query Transformation — The LangChain implementation
5. Microsoft — Improving Retrieval Quality with Query Optimization — A use case with Azure
6. HyDE Paper — Precise Zero-Shot Dense Retrieval without Relevance Labels — The original HyDE paper (a preview of capsule 06)

Estimated time: 25-30 minutes Next: 03-query-expansion.md