Silent MCP tool failures can make agents look productive while quietly increasing token usage and wait time. By combining Unity Gateway tracing with Genie One, Databricks engineers turned that hidden waste into a ranked, fixable bug list, identifying an estimated $1.2 million per year in wasted spend and lost productivity.
Key highlights
- Tool failures create hidden costs: Across one agent fleet, seven MCP-server bugs produced 1,409 errors per day, an estimated $499K in annual token costs, and about 12,000 hours of annual agent wait time.
- Tracing makes agent behavior observable: Unity Gateway automatically records OpenTelemetry traces for MCP tool calls, including tool names, arguments, errors, token counts, latency, and session IDs. The unified trace table makes it possible to connect wasted spend to specific tools and failures.
- Genie One turns traces into answers: Instead of spending hours exploring schemas and writing SQL, the team asked natural-language questions about recurring errors, recovery time, token cost, and wait time. Genie One returned a prioritized list of what to fix.
- Design tools for how models actually call them: LLMs make reasonable guesses when tool signatures are ambiguous. Tools should handle common variations gracefully, provide useful errors, and avoid crashing on inputs that are semantically valid but shaped differently than expected.
- The fix loop can be fast: Once the highest-impact failures and their causes were clear, coding agents helped apply the fixes. The full cycle, find, quantify, and fix, took about an hour.
๐ Read the full post here ๐
