I ran the same 24 coding tasks through three arms: fixed Haiku, fixed Opus and Databricks-managed selection. The managed-selection arm passed all 24 tasks. Its 24 routing records also said task profiling was unavailable.
That is not a contradiction. The service selected Opus 5 as a conservative fallback, and the worker met every task's acceptance contract. A dashboard showing only task success would have missed the reported profiling failure.
A passing task verifies the tested acceptance contract. It does not verify that the intended routing policy ran.
The evidence comes from those 72 runs and a separate route-only diagnostic in one environment. Treat it as a case study, not a verdict on Smart Routing across customers or future versions.
How I ran the experiment
The corpus contained 24 small coding tasks: eight each labeled easy, medium and hard. These were author-assigned labels, not calibrated difficulty scores. Tasks covered localized fixes, SQL, configuration, asynchronous behavior and changes across modules. Each arm ran each task once, starting from the same source commit in an isolated Git worktree.
The worker was Claude Code through the Databricks Unity Gateway CLI. Two arms pinned the main worker model. The third requested a decision from Databricks-managed task_v3 routing.
The routed arm ran in two stages. The unattended CLI routing hook returned a model selection without continuing the coding task, so the runner captured that decision and launched a separate worker with the selected model pinned. This measured managed selection followed by worker execution. It did not measure native end-to-end Smart Routing or recursive subagent routing.
| Arm | Tasks meeting the acceptance contract | Recorded main-model selection |
| Fixed Claude Opus 5 | 24/24 | Opus 5 pinned |
| Fixed Claude Haiku 4.5 | 23/24 | Haiku 4.5 pinned |
| Managed selection, then pinned worker | 24/24 | Opus 5 for all 24; every rationale reported profiling unavailable |
Success required the deterministic verifier to pass, required files to exist and forbidden files to remain unchanged. The Haiku failure illustrates why the last condition matters: its tests passed, but the agent modified a protected test file. The run remained a failure.
One observation per task cannot establish quality equivalence. The routed results also cannot establish adaptive-routing savings because the intended task profiler was unavailable throughout the recorded routing decisions.
HTTP 200 was not the health check
Databricks describes Smart Routing as selecting a suitable lower-cost model for each task. Its documented prerequisites include an account-level preview, candidate-model access and a supported region. It is a beta feature, with models and APIs subject to change.
In the separate route-only diagnostic, the managed selection endpoint returned HTTP 200, selected Opus 5 and explained that it had used a conservative route because profiling was unavailable. The matrix runs independently retained the CLI's selected model and rationale. All 24 reported the same profiling limitation.
That evidence separates three questions:
- Did the request succeed? The routing endpoint returned a response.
- Did task profiling succeed? The response explicitly said it did not.
- Did the worker satisfy the task contract? The evaluator checked that independently.
A fallback can preserve service availability while preventing an experiment from measuring its intended treatment. Continuing the task may be a reasonable application policy. Counting that task as evidence of healthy adaptive routing is a different decision.
The selection distribution alone would not establish a fault. A healthy router might reasonably choose one model for an entire corpus. The explicit failure rationale is what distinguishes this case. These observations do not establish the internal cause of that failure or a permission change that would fix it.
Give a benchmark an explicit fallback policy
Before launching an expensive task matrix, define what admits a run. A useful gate distinguishes healthy, fallback, failed and unverified decisions.
The following is pseudocode for that gate, not an official Databricks SDK interface:
- Request failed or response malformed: save the evidence and stop.
- Response reports fallback or profiler failure: save the evidence and stop.
- Selected model is outside the allowed candidate set: save the evidence and stop.
- Response does not match a reviewed successful-routing contract: mark it unverified and stop.
- Otherwise, admit the worker run.
The gate also needs a positive check. Omitting a known error string does not make a response healthy. If a beta response changes and the harness no longer understands it, the harness should retain it for review rather than silently classify it as success.
In this harness, a detected routing failure now stops execution before the solver, preserves the attempt and returns a nonzero exit code. A separate preflight leaves unfamiliar health metadata unverified. A successful profiling contract was not observed in this study, so this is a fail-closed design recommendation, not a validated drop-in health check for every Smart Routing version.
For an availability-focused evaluation, continuing through fallback can be intentional. Predeclare that treatment and report fallback outcomes separately. Do not remove those rows after seeing their quality or cost.
Audit what the agent actually called
Pinning the main model did not make the whole harness single-model. The gateway export for this experiment contained 569 unique recorded invocations. Both Opus-based arms included 24 Haiku invocations alongside 127 Opus invocations. The fixed-Haiku arm contained 267 Haiku invocations.
The attribution check matched the requester-filtered export to the serial run windows. All 72 runs had matching records, with no unmatched or multiply matched records in that export. This checks the exported records; it does not prove that every internal platform operation was exposed as a gateway invocation. The attribution also depends on controlling other traffic from the same requester.
Databricks documents an invocation_id for individual inference calls, destination-model fields and request tags in system.ai_gateway.usage. Multiple invocations can share a request ID. Record model usage at that granularity instead of attributing everything to the model pinned on the command line.
For a new experiment, attach a run tag wherever the integration supports it and verify that it reaches telemetry. Check export pagination, truncation, duplicates and run coverage before calculating totals. Missing telemetry should remain missing, not become zero cost.
Keep the monetary accounting separate too. A CLI token-price estimate, gateway-derived model estimate, billable-usage valuation and final invoice can cover different calls, rates and scopes. Databricks billing records arrive asynchronously, so an empty immediate query is not evidence of free execution.
This article makes no savings claim. Neither failed profiling nor differing accounting scopes support one.
The result worth carrying into your own system
In this experiment, the router reported a profiling failure and selected a conservative fallback. The fallback worker still passed all 24 tasks.
That result does not measure the quality or savings of a healthy Smart Routing classifier. It also does not show that fallback is a bad availability policy, or that either model is generally better for software engineering.
The practical fix is small: record routing health as its own result. Save it beside task acceptance and reconcile the calls that followed. For a benchmark, stop before solver execution when the intended selection policy cannot be verified. For an application that permits fallback, make that state visible and measure its cost separately.
You can then answer both questions a routing rollout needs: did the work get done, and did the intended routing policy run?
