Stop Asking an LLM Judge Questions Your Code Can Answer

[ivanvyd]

Suppose a RAG application returns this:

{
  "response": "You can return the item within 90 days.",
  "citations": ["returns-policy"]
}

The cited document allows returns within 30 days.

A citation-presence check should pass this response. A check that the source supports the answer should fail it.

You can inspect the citations field in code. Assessing whether a free-form answer is supported by a document is a reasonable job for an LLM judge. Keep those scores separate so a present citation cannot be mistaken for a supported claim.

Define the requirement first

For this example, the requirement is:

Every response in this evaluation dataset must contain a non-empty list of nonblank citation IDs.

That gives us a structural check. Assessing whether every factual claim has a supporting citation would require identifying those claims and comparing them with the evidence.

MLflow supports @scorer functions that consume application outputs and return Boolean results. Here is the structural check, using the documented code-based scorer API:

from mlflow.genai.scorers import scorer


@scorer
def has_citation(outputs: object) -> bool:
    """Check for a non-empty list of nonblank citation IDs."""
    if not isinstance(outputs, dict):
        return False

    citations = outputs.get("citations")
    return (
        isinstance(citations, list)
        and len(citations) > 0
        and all(
            isinstance(citation_id, str) and citation_id.strip()
            for citation_id in citations
        )
    )

A length check alone would accept "returns-policy" or [None]. This version rejects both.

The function checks only the citation field. It does not validate the whole response schema, resolve the IDs, or read the documents. It also assumes citations are required for every record in this dataset. An application that permits uncited greetings or refusals needs a different contract.

With an MLflow tracking destination and experiment configured, you can evaluate three toy records:

import mlflow

data = [
    {
        "inputs": {"question": "What is the return window?"},
        "outputs": {
            "response": "You can return the item within 90 days.",
            "citations": citations,
        },
    }
    for citations in (["returns-policy"], [], "returns-policy")
]

results = mlflow.genai.evaluate(
    data=data,
    scorers=[has_citation],
)

MLflow accepts precomputed inputs and outputs, so this example does not need a predict_fn. See the mlflow.genai.evaluate API reference.

The predicate returns True, False, and False for those citation values. The incorrect 90-day claim passes through untouched because this scorer never examines it.

Check citation structure and support separately

For the opening example, I would keep three requirements separate:

Requirement	Evidence needed	Evaluation approach
Citation IDs have the required structure	Application output	Code
Each ID refers to a document retrieved for this request	Output and recorded retrieval results	Code
The answer’s claims are supported by the retrieved material	Answer and document content	Semantic evaluation

The second check should use IDs recorded by the retrieval layer. Comparing citations against another list generated by the model would leave both sides of the check dependent on its output.

The third requires reading the evidence. A document can exist and have been retrieved while still contradicting the answer.

MLflow’s RetrievalGroundedness judge assesses support from the supplied context. Its documented trace-based workflow requires at least one RETRIEVER span, with inputs and outputs on the root span. Adding it to the output-only example above would leave it without the required retrieval evidence. See the groundedness judge requirements.

Groundedness has a further limit: an answer can faithfully repeat an outdated or incorrect source. Evaluating support from that source does not independently verify that the source is true.

Check what the Python function actually does

MLflow uses code-based scorer for a Python-defined evaluator. That function can call an LLM, wrap a built-in judge, or run other custom logic. The custom-scorer documentation covers these options.

The @scorer decorator therefore tells you nothing about whether the result is deterministic.

For the checks discussed here, I mean explicit rules applied to fixed, recorded evidence without a model call. Schema validation, required fields, allowlists, argument constraints, and comparisons against known expected values fit that description.

Those rules still need to measure something useful. A regex that finds a URL may return the same result every time, but naming that result answer_verified would overstate what it establishes.

Tool names are only part of the evidence

Suppose a test case requires:

get_customer → issue_refund

The intended requirement might include ordering, matching customer IDs, and a successful transaction. A set-membership check establishes only that both tool names appear.

MLflow traces contain requests, responses, tool parameters, timing, and other execution data for instrumented spans. Use the fields needed by the requirement. A list of names cannot establish that the customer lookup finished before the refund started or that the refund used the correct customer ID.

To confirm the refund completed, inspect a trustworthy transaction result or the system that records it. A recorded invocation alone establishes an attempt.

Missing telemetry also needs care. An absent span might mean the tool never ran, or that instrumentation failed to capture it. Unless coverage is known to be complete, report missing evidence rather than silently returning a pass.

Now consider a broader question:

Given the conversation and refund policy, should the agent have issued a refund?

Some cases reduce to an explicit rule such as “refunds above this amount require approval.” That rule belongs in code. Other cases involve interpreting the conversation or an ambiguous policy exception, where an LLM assessment and human review may be appropriate.

Choose the evaluator based on the decision you need to check, rather than assigning all tool-use or policy questions to a judge.

Keep the individual scores

A single prompt that checks formatting, citations, tool use, groundedness, relevance, and policy compliance leaves several possible explanations for FAIL. Separate assessments make those failures easier to investigate.

For a RAG application, I would keep has_citation and citation-ID checks alongside RetrievalGroundedness. RelevanceToQuery assesses whether the response answers the request, while Guidelines can evaluate specified natural-language criteria. The built-in judge documentation describes those criteria and their requirements.

A release gate can combine the results while retaining each assessment for debugging. Record skipped checks separately, too. A skipped groundedness check supplies no evidence of a pass.

Before relying on a semantic judge to block a release, I would compare its decisions with human-reviewed examples. The choice of evaluator still needs validation against the cases it will judge.

Enforce rules before the action

Consider this requirement:

refund amount must be greater than zero

Validate it before executing the refund. A scorer can inspect recorded attempts afterward, but runtime validation has to happen on the path that executes the operation.

Databricks production monitoring evaluates a configurable sample of incoming traces and attaches assessments. The documentation describes it as a Beta feature. Those assessments can help identify failures in recorded traffic; they do not stop the operation before it happens.

Reusing development scorers in production also has practical requirements.

For Databricks-managed monitoring, custom scorers must use @scorer and be defined and registered from a Databricks notebook. Custom class-based Scorer subclasses are not supported. See the custom-scorer restrictions.

The functions must be self-contained, with required imports inside the function body. After registration, call .start() with a sampling configuration, as described in the monitoring setup.

Check the available data before reusing a scorer. Registered production scorers obtain inputs and outputs from traces; the Databricks scorer reference says expectations is unavailable. A scorer that relies on hand-labeled expected tools in an offline dataset will need changes before it can evaluate live traffic.

Support also differs by deployment. The open-source MLflow custom-scorer documentation says code-based scorers are unsupported by its automatic evaluation feature. Check the deployment you are using before assuming a Databricks-managed monitoring example will work unchanged.

Give each metric a precise meaning

Before adding a scorer, write down its pass condition and the evidence it needs. A required field, known expected argument, or explicit policy limit usually gives you a direct check. Assessing whether a free-form answer addresses an ambiguous request may justify a judge.

In the return-policy example, has_citation=True means the citation field meets the contract. Whether the policy supports a 90-day return window remains a separate question, with a separate assessment.