Hello everyone,
Databricks’ Adaptive Instructed-Retriever announcement describes a model that learns when to stop searching. How would you test whether it stops after finding a general policy but before finding the exception that changes the answer?
I’d consider an offline test using questions with reviewed supporting evidence. Where the implementation allows it, continue early-stopped runs from the same history with a small extra search budget. Keep the model, corpus and final passage count fixed, then measure how often further search recovers missing evidence.
I’d measure evidence coverage separately from answer quality. An unsuccessful extra search wouldn’t prove the original retrieval was complete, either.
Has anyone tried something similar? What baseline or metric helped you judge whether the latency savings justified the risk of missing evidence? I am very eager to get your thoughts.
