topic Re: Bug? Genie Space benchmark evaluates "Score reason" wrongly as "Empty result" in Data Engineering

Bug? Genie Space benchmark evaluates "Score reason" wrongly as "Empty result" on agent mode

KriStromNAF — Wed, 01 Jul 2026 12:17:46 GMT

First some context: I'm setting up a Genie Space and have multiple benchmark questions with business validated SQL ground truths defined. Currently no configuration defined, apart from metadata and SQL joins.

I haven't found any other way of reporting a possible bug, hence making a discussion here.

The problem: When running the benchmark on Agent mode, all questions get a model answer and some output, but still the evaluation conclusion returns "Score reason": "Empty result" even though there is some result values. Sometimes, the output is actually correct and matching the ground truth, but still evaluated as empty result. This does not happen when running the benchmarks in chat mode.

Anyone else encountered the same issue? Is it expected behaviour?

Re: Bug? Genie Space benchmark evaluates "Score reason" wrongly as "Empty result"

iyashk-DB — Thu, 02 Jul 2026 07:25:01 GMT

Hi, this looks like a real quirk in agent mode benchmark scoring, not something you're doing wrong on the setup side.

Chat mode benchmarks grade by comparing Genie's generated SQL result set against your SQL ground truth, so "Empty result" there means the query genuinely returned nothing. Agent mode works completely differently: there's no SQL comparison at all, an LLM judge reads the whole agent report (text summary, tables, citations) and grades it against your Evaluation note. Since agent mode responses come back as a report rather than a clean result set, it's plausible the judge is failing to parse a result out of that report format in some cases and defaulting to "Empty result" even when the report clearly has an answer in it. That would explain why you're seeing it on outputs that already match ground truth.