Greetings @lkt1 , Good set of questions here, and you're already thinking about this the right way: reuse the data you've got rather than bolt on redundant scorers. You're really asking two things, and they have different answers, so let me take them one at a time.
Short version up front: the Evaluation runs overview table itself isn't documented as configurable for arbitrary columns, but everything you're after is reachable, and you don't need a new latency scorer to get the average.
Percentiles and medians of your score
This is the direct one. Scorers take an aggregations argument, and the default is mean only, which is why you're seeing a single number. You can ask for more when you define the scorer:
@scorer(aggregations=["mean", "median", "p90"])
The built-in options are min, max, mean, median, variance, and p90. If you need something else (p50, p95, p99), pass a callable that takes a list of values and returns one number. Each aggregation gets logged as its own run metric, roughly scorer_name/median and scorer_name/p90.
One catch to save you some trouble: these are computed when mlflow.genai.evaluate() runs, so the UI won't go back and add a median to a run that only logged a mean. If you'd rather not re-run, pull the per-row feedback with mlflow.search_traces() and compute the stats in pandas.
Average latency (you already have it)
You don't need a new scorer for this one. An evaluation run auto-logs the agent metrics (latency and token counts) as run-level aggregates, so the average latency is already sitting there as data. If you specifically want median or p90 latency, add a small custom scorer that returns the trace execution time and set aggregations=["mean", "median", "p90"] on it, same pattern as above.
Getting them into the overview
Once those values are logged as run metrics, take a gander at the Columns control on the Evaluation runs table (the same button in your screenshot). I'll be straight with you: the docs don't spell out exactly which metrics that picker exposes for this particular view, so confirm it actually surfaces the latency and percentile columns rather than taking my word for it.
If you want this for reporting or a dashboard
The MLflow system tables are the better tool for anything persistent. system.mlflow.run_metrics_history holds every metric logged on a run, and system.mlflow.runs_latest holds the aggregated min, max, and latest per metric, so you can query across runs in SQL and build an AI/BI dashboard on top. One thing to keep in mind: these tables reflect the metrics you logged, so they'll give you the median or p90 you configured above, not a percentile you never computed. Three practical caveats:
- They're in Public Preview.
- Access is admin-gated by default (an account admin grants
USE and SELECT on system.mlflow).
- Data only goes back to September 2, 2025.
The takeaway
Configure the aggregations you want at evaluation time, use the auto-logged latency for the average, and reach for the system tables when you need flexible reporting. One prerequisite before any of this: make sure you're on mlflow[databricks]>=3.1, since the aggregations API and the current GenAI eval UI came in with the MLflow 3 integration.
Hope that helps
Regards, Louis
