Reformulating and Retrying the Query

Description

Lesson 02 closed Question 2 with an observation left hanging: search_docs("What is Reservo's cancellation policy?", k=3) gets the expected doc_id (cancellation-policy) into first place, but the winning chunk — cancellation-policy-003, the "Related Policies" section — is only a cross-reference sentence ("see refund-policy... see no-show-policy..."), not the real text of the cancellation window. A monolithic pipeline would settle for that result because the doc_id "was right." An agent that actually reads what it retrieved doesn't settle: it recognizes the chunk doesn't answer the question, reformulates the query, and tries again.

This lesson builds that second half of the cycle. You're going to write a checker — real, executed — that detects when a chunk is predominantly a reference to other documents rather than content of its own, and you're going to see, with two real search_docs calls, how a second query with different words finds the chunk the first one missed.

Connection to the module

This lesson builds directly on Lesson 02's Question 2: same case, same first result, but now with the missing piece — what to do when that result isn't enough. It's the foundation for Lesson 04 (multi-hop), which generalizes this pattern to questions that need several independent searches, not just a second attempt at the same question.

Analogy: the folder that only says "see also"

The module introduction's researcher asks the archive for the "cancellation policy" folder. The archivist brings a folder — but opening it, all that's inside is a note: "for refunds, see the refunds folder; for no-shows, see the no-shows folder." The folder technically matched what was asked for, but it doesn't contain the answer — it's a pointer index, not the content. A researcher who settles for that note and reports it as if it were the complete policy is failing, even though they "found something with the right name." A real researcher notices the difference, and goes back to the desk with a more specific question: not "what's the cancellation policy?" but "how many hours in advance can I cancel free of charge in pro mode?" — a question that can no longer be answered with a simple pointer note.

An executable checker: is this chunk just a cross-reference?

Reservo's canonical corpus deliberately has chunks that mention other doc_id values inside backticks — you saw this since Module 2 — to simulate how a real company's documents reference each other. That's genuinely useful for a human reader browsing the whole archive, but it's a warning sign when that specific chunk is the only result the agent is going to read. Let's build a simple detector, based on that same feature of the text:

import re

REF_RE = re.compile(r"`[a-z]+(?:-[a-z]+)+`")


def looks_like_cross_reference(text, min_refs=2):
    """True if the text mentions 2 or more doc_id-style identifiers inside
    backticks (`refund-policy`, `no-show-policy`, ...) -- a signal that
    the chunk is a pointer note, not real content."""
    return len(REF_RE.findall(text)) >= min_refs

REF_RE looks for the exact pattern the corpus uses to write a reference to another document: one or more lowercase words separated by hyphens, inside backticks — the same shape as `refund-policy` or `no-show-policy`. min_refs=2 is a deliberate choice: a chunk that mentions a single doc_id in passing (common, and not necessarily a problem) doesn't get flagged; two or more mentions in the same short chunk is the real signal that the chunk exists to point elsewhere, not to answer on its own.

Worked example: the checker against the real first result

Picking back up exactly Lesson 02's search:

from search_docs_tool import search_docs

results = search_docs("What is Reservo's cancellation policy?", k=3)
for r in results:
    flag = looks_like_cross_reference(r["text"])
    print(f"  {r['chunk_id']:28s} score={r['score']:<7} cross_reference={flag}")
    print(f"    text: {r['text']!r}")

What to expect:

  cancellation-policy-003     score=10.734  cross_reference=True
    text: 'See `refund-policy` for what happens to the money once a cancellation is processed, and `no-show-policy` for what happens if you simply do not show up without cancelling.'
  refund-policy-000           score=7.217   cross_reference=False
    text: "A refund applies when a booking is cancelled within the cancellation window for the member's tier, or when Reservo cancels a booking due to a facility issue on Reservo's side."
  no-show-policy-000          score=6.364   cross_reference=False
    text: 'A no-show is a booking where the member never checks in during the reserved hours and never cancelled beforehand. This is different from a late cancellation, which is covered in `cancellation-policy`.'

There's the signal, executed: the chunk that won first place (cancellation-policy-003) contains exactly two backtick-wrapped references — `refund-policy` and `no-show-policy` — and looks_like_cross_reference flags it True. The third result (no-show-policy-000) mentions `cancellation-policy` just once — below the threshold, correctly not flagged, because a single passing mention doesn't turn an entire chunk into a simple pointer note.

Concept: the agent decides to reformulate

With the signal executed and in hand, the next step — deciding to reformulate, and with what words — is conceptual: the reformulation itself is a model decision, not something a checker can generate automatically.

[model·concept]
search_docs's first result has the correct doc_id
(cancellation-policy), but looks_like_cross_reference flags it as a
pointer note, not the policy's real content. The original question
("What is Reservo's cancellation policy?") is too general -- it
matches the sentence that only NAMES the related policies. I'll
reformulate, targeting the specific data the original question
needed: how many hours in advance you can cancel free of charge in
pro mode.
decision -> use_tool "search_docs" with {query: "How many hours in
advance is a pro tier cancellation free of charge?", k: 3}

The reformulation doesn't change topic — it's still about the pro cancellation window — but it changes the level of specificity: from a general question a cross-reference can "answer" superficially, to a concrete question only the chunk with the real number can satisfy.

Worked example: the second search, executed

retry_results = search_docs("How many hours in advance is a pro tier cancellation free of charge?", k=3)
for r in retry_results:
    flag = looks_like_cross_reference(r["text"])
    print(f"  {r['chunk_id']:28s} score={r['score']:<7} cross_reference={flag}")

What to expect:

  cancellation-policy-001     score=15.138  cross_reference=False
  no-show-policy-000          score=8.757   cross_reference=False
  cancellation-policy-000     score=7.715   cross_reference=False

The reformulation worked, in both ways that matter. First, the winning chunk changed: cancellation-policy-001 — "Pro Tier Cancellation Window," the real text: "Pro members get a shorter, friendlier window: cancellations up to 4 hours before the reserved start time are free of charge..." — replaced the first attempt's pointer note. Second, looks_like_cross_reference confirms False on all three results: none is a simple cross-reference this time. And the margin is much wider than the first search: 15.138 against 8.757 (almost double), versus 10.734 against 7.217 in the first attempt — an additional signal, though not conclusive on its own, that this second query found something more specific.

The complete pattern, visualized

query 1: "What is Reservo's cancellation policy?"
    │
    ▼
search_docs -> top1: cancellation-policy-003 (only points to other policies)
    │
    ▼
looks_like_cross_reference(top1.text) -> True   (REAL signal, executed)
    │
    ▼
[model·concept] "this doesn't answer the question, reformulating"
    │
    ▼
query 2: "How many hours in advance is a pro tier cancellation free of charge?"
    │
    ▼
search_docs -> top1: cancellation-policy-001 (the real text, "4 hours")
    │
    ▼
looks_like_cross_reference(top1.text) -> False   (real content, not a note)

Two calls to search_docs, both real and both within the same logical conversation turn — from the point of view of the agent's history (the messages you already know from agent-fundamentals-and-tool-calling Module 4), they're two trips around the loop: one tool decision, one result, a second tool decision with a different query, a second result, and only then the final answer.

Why this isn't "search until it works"

It's worth being precise about what this pattern is and isn't. It isn't a while that retries the same query until the score clears some magic number — that would be brute force, not reformulation. It also isn't retrying with a random query until something hits. It's an informed decision: the agent reads the result's text (not just the score), identifies why it doesn't work — in this case, with a concrete, executable signal: it's a pointer note, not content — and builds a second query that specifically targets the missing data. looks_like_cross_reference doesn't decide the reformulation for the agent; it gives the agent (concept) a concrete, verifiable reason not to settle for the first result.

Common mistakes

1. Retrying without changing the query. Running search_docs twice with exactly the same question text produces, with a deterministic index like this guide's, exactly the same result — zero gain. The reformulation has to change the query's vocabulary, not just repeat the call.

2. Confusing "the doc_id was right" with "the chunk answers the question." As you saw in Module 3, a correct doc_id in the top-1 doesn't guarantee the specific chunk is useful — it can be, as in this case, a cross-reference section from the same correct document. Checking the doc_id alone isn't enough; you have to look at the text.

3. Reformulating without limit, with no attempt cap. Nothing in this lesson stops a poorly designed agent from reformulating indefinitely if no query finds anything good. The loop's iteration cap (max_iterations, already established in agent-fundamentals-and-tool-calling Module 4) still applies here unchanged — reformulating consumes one trip around the loop, like any other tool call.

4. Thinking looks_like_cross_reference is a general test for "bad chunk." It's a checker specific to a concrete pattern in this corpus (mentions of other doc_id values inside backticks). A chunk can be useless for a question for many other reasons — wrong topic, too generic, truncated mid-idea — that this specific checker doesn't detect. It's a useful signal, not a complete quality guarantee.

Exercises

Exercise 1: Apply the checker to a new result (Easy)

Without running anything: given the text "No-shows are never refunded, regardless of membership tier. See \no-show-policy` for the full no-show rules and the fee that applies instead."(chunkrefund-policy-002), how many backtick-wrapped references does it contain? Would looks_like_cross_referenceflag itTrueorFalse`? Confirm by running it.

text = "No-shows are never refunded, regardless of membership tier. See `no-show-policy` for the full no-show rules and the fee that applies instead."
print(looks_like_cross_reference(text))

False

Exercise 2: Reformulate a different query and confirm the improvement (Medium)

Starting from search_docs("What fee applies if a member does not show up for a booking?", k=3), confirm the top-1 is a cross-reference. Reformulate with search_docs("Does a no-show get charged the full reserved price?", k=3) and confirm the second attempt finds no-show-policy-001 without being a cross-reference.

first = search_docs("What fee applies if a member does not show up for a booking?", k=3)
for r in first:
    print(r["chunk_id"], r["score"], looks_like_cross_reference(r["text"]))

print()

retry = search_docs("Does a no-show get charged the full reserved price?", k=3)
for r in retry:
    print(r["chunk_id"], r["score"], looks_like_cross_reference(r["text"]))

cancellation-policy-003 17.143 True
no-show-policy-003 11.418 False
refund-policy-002 9.844 False

no-show-policy-001 13.335 False
refund-policy-002 7.809 False
no-show-policy-000 6.636 False

Exercise 3: Design a reformulation cap (Hard)

Write a function search_with_retry(query, retry_query, k=3, max_attempts=2) that calls search_docs with query; if the top-1 passes looks_like_cross_reference, retries once with retry_query; returns the result of whichever attempt worked (or the second one, if neither did). Run it with the worked example's pair of queries and confirm it returns the second search's results.

def search_with_retry(query, retry_query, k=3, max_attempts=2):
    """Search with `query`; if the top-1 is a cross-reference, retry
    ONCE with `retry_query`. max_attempts bounds the number of calls to
    search_docs, the same way max_iterations bounds the agent's complete loop."""
    attempts = 0
    results = search_docs(query, k=k)
    attempts += 1

    if results and looks_like_cross_reference(results[0]["text"]) and attempts < max_attempts:
        results = search_docs(retry_query, k=k)
        attempts += 1

    return results, attempts


final_results, attempts = search_with_retry(
    "What is Reservo's cancellation policy?",
    "How many hours in advance is a pro tier cancellation free of charge?",
)
print(f"attempts used: {attempts}")
for r in final_results:
    print(f"  {r['chunk_id']:28s} score={r['score']}")

attempts used: 2
  cancellation-policy-001     score=15.138
  no-show-policy-000          score=8.757
  cancellation-policy-000     score=7.715

Summary and next step

• The correct doc_id in the top-1 doesn't guarantee the chunk answers the question — Modules 2/3 already hinted at this; this lesson made it explicit with a real checker: looks_like_cross_reference, which detects chunks that only name other policies inside backticks.
• We ran the complete pattern: search_docs with the original query returns a cross-reference (True); the model (concept) reformulates, targeting the specific missing data; the second search_docs, executed, brings back the real chunk (False), with a much wider margin than the first attempt.
• Reformulating isn't brute force: it's an informed decision, based on reading the result's text, not just its score or doc_id.
• The loop's iteration cap (agent-fundamentals-and-tool-calling Module 4) still applies: every reformulation consumes one trip around the loop, and the bounded while/for doesn't change shape just because one of the tools is a retrieval one.

Next lesson: 04 — Multi-hop retrieval. What happens when the question doesn't need to reformulate the same search, but instead needs two independent searches on two different topics?

Additional resources

1. Anthropic — Tool use (function calling) overview — The request-execute-result cycle this lesson repeats twice within the same logical conversation turn.
2. agent-fundamentals-and-tool-calling-guide, Module 4, Lesson 04 (The iteration cap) — the mechanism that keeps a reformulation from repeating without limit.
3. Python — Regular expressions (re) — the reference for re.compile/findall, used by REF_RE to detect the backtick-wrapped identifiers.
4. embeddings-deep-dive-guide — why a real semantic search could, in principle, tell "cancellation window" apart from "see also" without needing an explicit second query; this guide uses lexical BM25, which does need this reformulation pattern.