How do you test whether a retriever stops too early?

ivanvyd — Fri, 11 Sep 2026 12:12:48 GMT

Hello everyone,

Databricks’ Adaptive Instructed-Retriever announcement⁠ describes a model that learns when to stop searching. How would you test whether it stops after finding a general policy but before finding the exception that changes the answer?

I’d consider an offline test using questions with reviewed supporting evidence. Where the implementation allows it, continue early-stopped runs from the same history with a small extra search budget. Keep the model, corpus and final passage count fixed, then measure how often further search recovers missing evidence.

I’d measure evidence coverage separately from answer quality. An unsuccessful extra search wouldn’t prove the original retrieval was complete, either.

Has anyone tried something similar? What baseline or metric helped you judge whether the latency savings justified the risk of missing evidence? I am very eager to get your thoughts.

Re: How do you test whether a retriever stops too early?

DoTA — Sun, 13 Sep 2026 09:21:20 GMT

This is a good framing question. A few things that have worked well for evaluating early-stopping retrieval specifically:

Build a small labeled set where each query has a known minimal sufficient evidence set (the specific chunks/passages a correct answer actually needs, not just "the right document"). That label is what everything else below is measured against, so it's worth the upfront cost even for 30-50 queries.

Then track two separate numbers per query, not one blended score:

1. Evidence coverage at the stopping point: did the retrieved set contain the full minimal evidence set, yes/no (or fraction). This isolates retrieval quality from generation quality.

2. Answer correctness at the stopping point vs. answer correctness with an uncapped/exhaustive search budget as the ceiling. The gap between these two is the actual cost of stopping early, in outcome terms.

Your instinct to continue early-stopped runs from the same history with extra budget is the right diagnostic, I'd frame it as: how much does evidence coverage improve per additional hop past the current stopping point? If the marginal coverage gain flattens quickly after the stop, that's real evidence the policy is well-tuned, not just an absence of proof. If it keeps climbing, you're leaving recall on the table.

The distinction you're drawing between evidence coverage and answer quality also matters in the other direction: log cases where extra search added evidence but the answer didn't change (early stop was fine in practice even if technically incomplete) separately from cases where it changed a wrong answer to a correct one (that's your real miss rate). The first case is much more common than people expect and conflating the two overstates how risky early stopping actually is.

For the latency/cost side of the tradeoff: plot accuracy (or evidence coverage) against search budget (hops, tokens, or latency) across a few candidate stopping policies, including "always max budget" and "fixed shallow" as baselines. You're looking for the elbow in that curve, the point where extra budget stops buying meaningful accuracy. That elbow, not a single fixed threshold, is usually the more defensible answer when someone asks "why did you pick this stopping point," since it's tied to a measured diminishing-returns point rather than a guess.

topic How do you test whether a retriever stops too early? in Generative AI

How do you test whether a retriever stops too early?

Re: How do you test whether a retriever stops too early?