Handling zero results

Description

search_docs can return an empty list — Module 3 (Lesson 07) already showed it with a query made of pure noise ("xyzxyzxyz nonsense query zzzqqq") and Module 2 (Lesson 06) already established it as a legitimate BM25 index result, not an error. This lesson builds the missing piece: what the final answer does when that happens. The underlying question is one of grounding — does the agent honestly admit it found nothing, or does it fill the silence with an answer that sounds plausible but comes from no document at all? — and the correct answer looks obvious until you run the real case. You're going to see, with evidence from your own terminal, that search_docs returning [] is actually the rare case: the vast majority of questions Reservo can't answer don't come back empty — they come back with low-score chunks that, without looking closely at the number, look like normal results.

Connection to the module

This lesson builds on Lesson 02 (citing the source): before asking what to cite, you need to decide whether there's anything to cite. Lesson 04 (relevance threshold) picks up exactly this lesson's uncomfortable finding — that [] is rare — and provides the tool that does cover the common case.

Analogy: "we don't have that" instead of a made-up answer

At any real help desk, there are questions whose honest answer is "we don't have that" — a customer asks about the building's gym hours, and Reservo has no gym and no policy about one. Someone at the desk who takes their job seriously says "we have no information about that" without hesitating. Someone who doesn't want to disappoint anyone might, instead, improvise something plausible based on what they know about similar buildings — and that's where the problem starts: the customer walks away with an answer that sounds like official Reservo policy, but that no one wrote or approved. This lesson's challenge isn't teaching how to say "we don't have that" — that's easy when the question is obviously outside the business. The real challenge is noticing that, in practice, the automatic desk almost never comes back completely empty-handed: it almost always brings back something, even if that something has nothing to do with what was asked — and confusing "I brought something back" with "I found the answer" is, in this job, the most expensive mistake of all.

handle_no_results: the minimal piece

from search_docs_tool import search_docs

NO_RESULTS_MESSAGE = "I couldn't find information about this in Reservo's documents."


def handle_no_results(hits):
    """If hits is empty, returns the honest 'we don't have that' message.
    If there's at least one result, returns None -- the answer continues
    its normal course (citing whatever search_docs did bring back)."""
    if not hits:
        return NO_RESULTS_MESSAGE
    return None

On a query made of pure noise, with no real corpus term at all:

hits = search_docs("xyzxyzxyz nonsense query zzzqqq", k=3)
print("hits:", hits)
fallback = handle_no_results(hits)
print("answer:", fallback if fallback else "(normal flow continues with the hits)")

What to expect:

hits: []
answer: I couldn't find information about this in Reservo's documents.

This is the clean case: search_docs matched no corpus term to the query, returned [], and handle_no_results triggers the honest message. No LLM is involved in this part — it's deterministic logic over a Python list — but it's the indispensable foundation for what follows: [model·concept] with claude-sonnet-5, if search_docs's tool_result arrives as [] and the system prompt instructs it to answer only with what the tools return, the expected answer is a variation of "I couldn't find information about this in the available documents" — not a generic answer based on the model's general knowledge about coworking memberships. An agent without that explicit instruction, receiving [], could instead fall back on what it "knows" about similar systems and answer confidently about something Reservo never documented — exactly the kind of ungrounded answer Module 4 (Lesson 06) measures with check_grounding for numbers, and that this lesson prevents one step earlier, in the very decision of whether there's a basis to answer at all.

The uncomfortable finding: [] is the rare case

Before assuming handle_no_results covers "the questions Reservo can't answer", it's worth testing it against questions genuinely outside the business — not random noise, but real questions, with real grammar, about topics Reservo simply doesn't document:

off_topic_queries = [
    "What is the weather forecast for tomorrow?",
    "How do I reset my email password?",
    "Can I bring my dog to the building?",
]

for q in off_topic_queries:
    hits = search_docs(q, k=3)
    print(f"{q!r}")
    print(f"  hits={len(hits)}  ", [(h['doc_id'], h['score']) for h in hits])

What to expect:

'What is the weather forecast for tomorrow?'
  hits=3   [('cancellation-policy', 6.131), ('booking-faq', 4.843), ('cancellation-policy', 4.732)]
'How do I reset my email password?'
  hits=3   [('wifi-and-equipment-faq', 5.382), ('wifi-and-equipment-faq', 3.334), ('cancellation-policy', 2.548)]
'Can I bring my dog to the building?'
  hits=3   [('studio-room-manual', 3.658), ('booking-faq', 3.512), ('payment-methods-faq', 3.466)]

None of the three questions — about the weather, about an email password, about bringing a pet — has anything at all to do with Reservo's documents. And yet, none returns []. All three bring back three results, with scores in a range (2.5 to 6.1) that, at a glance, is indistinguishable from the range of scores the genuinely relevant anchor queries also produce. handle_no_results, as written, doesn't trigger the honest message in any of these three cases — to the system, "there are results" is indistinguishable from "there are relevant results".

The reason is the same one that's held up this guide's lexical honesty since Module 2: BM25 doesn't need a query to be about the corpus to find matches — it just needs it to share words, and with no stopword removal, almost any well-formed English question shares something ("is", "the", "does", "can") with some chunk from a 57-fragment corpus. [] only shows up when the query shares no term at all, not even a functional one — the case of "xyzxyzxyz nonsense query zzzqqq", tokens that literally don't exist in any language of the corpus. A real, grammatically normal question almost never falls into that extreme case.

Why this matters more than it seems

Think through the consequences of relying only on handle_no_results to decide when to admit "we don't have that". If the criterion were "answer with the honest message only if hits is empty", the three questions above — weather, email password, pets — would pass the check and continue the normal flow: search_docs would hand them three real chunks, with scores that look normal, and nothing along the way would warn the agent that those chunks have no relation to the question. [model·concept] an agent receiving those three cancellation-policy/booking-faq chunks as the tool_result for a weather question, with no additional signal, runs a real risk of citing them as if they were relevant — "according to our documents..." — simply because they arrived as the result of a tool that did execute successfully.

This isn't a flaw in handle_no_results — it's its honest, precisely labeled limit: it covers the case of total absence of lexical match, which is real but infrequent. The much more common case — weak lexical match, with no real thematic relation — needs a different tool, one that looks at the score, not just the list's length. That's precisely this module's Lesson 04.

The combined flow, visualized

search_docs(query, k) -> hits
    │
    ▼
hits == []?
    │
    ├── YES -> handle_no_results(hits) -> honest message, done
    │
    └── NO -> hits has 1+ results
              │
              but: are they relevant, or do they just share
              functional words with the query?
              (this lesson does NOT answer that -- Lesson 04 does)

handle_no_results resolves the diagram's left branch with three lines of logic. The right branch — what to do with non-empty but low-confidence results — is deliberately the next lesson's job, not this one's.

Common mistakes

1. Thinking handle_no_results covers "every question Reservo can't answer". As the previous section showed with three real questions, none genuinely off-topic returned []. handle_no_results covers exactly one case: total absence of lexical match. It's a necessary piece, not a sufficient one.

2. Treating [] as a system failure that needs to be "fixed" by forcing some result. As Module 2 already established and Module 3 confirmed, an empty list is the index's correct, honest answer when there's no match at all — forcing a low-score result in its place would replace an honest "I don't know" with a potentially misleading answer, exactly the opposite of what this lesson aims for.

3. Assuming the NO_RESULTS_MESSAGE replaces the need for a system prompt instructing the model to admit when it doesn't know. handle_no_results is a deterministic safeguard for the literal [] case — it doesn't replace the explicit instruction, at the agent's prompt level, that the answer must be based only on what the tools returned. Without that instruction, a model receiving [] as a tool_result could still try to answer from its general knowledge.

4. Not distinguishing, in the system design, between "0 results" and "low-confidence results". These are two different failures with different evidence: the first is detected with len(hits) == 0; the second needs to look at each result's score. Treating them as the same problem — "if handle_no_results didn't trigger, everything's fine" — is exactly the mistake this lesson exposed with the three off-topic questions.

Exercises

Exercise 1: Confirm the clean case (Easy)

Run handle_no_results on search_docs("qwjkl zxcvb poiuy", k=3)'s result (another pure-noise query, different from the worked example's) and confirm the honest message triggers.

hits = search_docs("qwjkl zxcvb poiuy", k=3)
print("hits:", hits)
print("answer:", handle_no_results(hits))

hits: []
answer: I couldn't find information about this in Reservo's documents.

Exercise 2: Measure the real "zero results" rate on a mixed batch (Medium)

Build a list with the corpus's six anchor queries (relevant) plus the worked example's three off-topic questions (weather, password, pet) plus one pure-noise query. Run search_docs on all ten and count how many trigger handle_no_results. Does the result confirm or contradict this lesson's finding?

batch = [
    "What is the cancellation policy for Boardroom bookings?",
    "How much discount does the pro tier get?",
    "What equipment is in the Focus room?",
    "Can I get a refund if I didn't show up?",
    "What payment methods does Reservo accept?",
    "Is there wifi in the Lounge?",
    "What is the weather forecast for tomorrow?",
    "How do I reset my email password?",
    "Can I bring my dog to the building?",
    "xyzxyzxyz nonsense query zzzqqq",
]

zero_count = 0
for q in batch:
    hits = search_docs(q, k=3)
    fallback = handle_no_results(hits)
    if fallback:
        zero_count += 1
    print(f"{'[]' if fallback else f'{len(hits)} hits':8s}  {q}")

print(f"\n{zero_count}/{len(batch)} questions triggered handle_no_results ({zero_count/len(batch):.0%})")

3 hits    What is the cancellation policy for Boardroom bookings?
3 hits    How much discount does the pro tier get?
3 hits    What equipment is in the Focus room?
3 hits    Can I get a refund if I didn't show up?
3 hits    What payment methods does Reservo accept?
3 hits    Is there wifi in the Lounge?
3 hits    What is the weather forecast for tomorrow?
3 hits    How do I reset my email password?
3 hits    Can I bring my dog to the building?
[]        xyzxyzxyz nonsense query zzzqqq

1/10 questions triggered handle_no_results (10%)

Exercise 3: Design a more specific honest message (Hard)

NO_RESULTS_MESSAGE is a generic message, the same for any query. Write a function handle_no_results_verbose(query, hits) that, when hits is empty, returns a message including the original query in quotes — for example, so a user can confirm the system correctly understood what they asked. Run it on the worked example's noise query and discuss, in one sentence, a risk of including the user's literal query inside a response message.

def handle_no_results_verbose(query, hits):
    """Version of handle_no_results that echoes the original query, so
    the user can confirm the system understood the question correctly."""
    if not hits:
        return f'I couldn\'t find information about "{query}" in Reservo\'s documents.'
    return None


msg = handle_no_results_verbose("xyzxyzxyz nonsense query zzzqqq", [])
print(msg)

I couldn't find information about "xyzxyzxyz nonsense query zzzqqq" in Reservo's documents.

Summary and next step

• handle_no_results(hits) triggers an honest "we don't have that" message exactly when search_docs returns [] — total absence of lexical match with the corpus.
• The central, executed finding: [] is the rare case. Three genuinely off-topic questions (weather, email password, pets) all three returned non-empty results with scores indistinguishable at a glance from a relevant query — of a batch of ten questions, only 10% triggered the honest message.
• This result doesn't invalidate handle_no_results — it correctly places it as a necessary, but insufficient, piece for honestly handling "I don't know": it covers the extreme of total signal absence, not the far more common case of weak signal with no real thematic relation.
• [model·concept] an agent instructed to answer only with what the tools return, receiving [], admits it found no information; without that explicit instruction, it risks falling back on its general knowledge — the underlying reason this piece matters before you even get to generating the final answer.

Next lesson: 04 — Filtering irrelevant chunks with a threshold. The piece that does cover the common case: non-empty results, but with relevance too low to cite seriously.

Additional resources

1. production-rag-and-document-ingestion-guide, Module 3, Lesson 07 (search_docs run end to end) — this guide's first executed [] case, with the noise query this lesson picks back up.
2. production-rag-and-document-ingestion-guide, Module 2, Lesson 06 (The lexical limit: a synonym that doesn't match) — why [] is a legitimate index result, not an error to fix.
3. production-rag-and-document-ingestion-guide, Module 4, Lesson 06 (Grounding: anchoring the answer in the chunks) — check_grounding, the checker that confirms whether a number cited in the final answer has real backing in the history, the piece that acts one step after this lesson's decision.
4. Python — Truth value testing of empty sequences — the foundation of if not hits:, used in handle_no_results to detect the empty list.