3 weeks ago
I'm trying to work out what a sensible guardrail looks like for each agent when several of them share a few serving endpoints.
A plain token budget feels too blunt. An agent that retries a failing tool four times looks like a heavy user, while a cheap agent stuck in a loop can burn far more than its neighbours before anyone notices.
The options I'm weighing:
1. A hard daily cap per agent on tokens, through AI Gateway rate limits.
2. An alert on tool invocations per request, to catch loops and retry storms early.
3. An alert on cost per completed task against that agent's own 7-day baseline.
For those running agents in production on Databricks: which of these do you actually use, and which one did you regret? And is anything in the system tables or inference tables good enough to build that baseline from, or do you end up logging your own per-task numbers?
Happy to hear partial answers too, even just what you stopped doing.
3 weeks ago
Hi @DoTA,
Good question. I'd actually use all three of your options, but each one does a different job, and the first needs a small change.
system.ai_gateway.usage table has a request_id plus an invocation_id for each model call, so multi-turn agent calls can be grouped together (Reference). The older system.serving.endpoint_usage table has a usage_context map where you can pass your own identifiers, such as an agent name or task ID. These tables are good for showing who is using how much, but they don't know where a task begins or ends, or whether it finished.usage_context is a reasonable fallback, but it gives you cost per request, not cost per task.If this answer resolves your question, could you mark it as โAccept as Solutionโ? That helps other users quickly find the correct fix.
3 weeks ago
Short answer: I'd use all three, but at different layers. A per-agent rate limit is the hard backstop, a calls-per-request alert is the early warning for loops and retry storms, and cost per completed task against a 7-day baseline is a review metric, not a guardrail. What makes all three work is giving every agent its own identity (a service principal) and tagging every request with the agent and task, because both the rate limiter and system.ai_gateway.usage key on who called and which tags came with the call [1][2].
Unity Gateway rate limits are set per minute or per hour, on requests and tokens, at four scopes: the whole service, a default per user, a specific user or service principal, and a group [1][3]. There is no daily window, so a "daily cap" becomes an hourly token ceiling sized at a few times the agent's normal peak.
The relevant part of the model service config looks like this [3]:
Two gotchas. The limiter records usage after a response is sent, so concurrent bursts can briefly overshoot; treat it as a backstop, not an exact budget [1]. And if you are still on AI Gateway on serving endpoints (legacy), TPM limits can't be applied to endpoints that serve custom models or agents, so put the token limit on the foundation model endpoint the agent calls [4].
This is the one that catches the cheap agent stuck in a loop. In system.ai_gateway.usage, every inference call gets its own invocation_id, and multi-turn agent calls share a request_id [2]. If your tools are exposed as Unity Gateway MCP services, mcp_metadata.tool_name is logged too. That gives you calls per request straight from the table:
If your agent framework sends each turn as a separate HTTP request, request_id won't tie a task together. In that case send agent and task_id in the Databricks-Ai-Gateway-Request-Tags header and group by request_tags['task_id'] instead [5]. Alert when max_model_calls goes above a multiple of that agent's own p95.
The table is for alerting after the fact. To kill a loop while it runs, also enforce a max-steps or max-tool-calls limit inside the agent code.
Useful for spotting drift after a prompt or model change, but it fires after the money is spent. Two practical points:
Skip the single shared daily token budget; that's exactly the noisy-neighbour problem you describe. Start with one service principal per agent and an hourly token limit on each, plus a max-steps limit in code. Add the calls-per-request alert from the query above, and review cost per completed task weekly rather than alerting on it.
[1] Apply rate limits to model and MCP services | Databricks on AWS โ https://docs.databricks.com/aws/en/ai-gateway/rate-limits
[2] Track model usage | Databricks on AWS โ https://docs.databricks.com/aws/en/ai-gateway/usage-tracking
[3] Model Service | Databricks API Reference โ https://docs.databricks.com/api/ai-gateway/v1/model-service
[4] Configure governance on model serving endpoints (legacy) | Databricks on AWS โ https://docs.databricks.com/aws/en/ai-gateway/configure-ai-gateway-endpoints
[5] Request tagging | Databricks on AWS โ https://docs.databricks.com/aws/en/ai-gateway/request-tagging
[6] Set up production monitoring | Databricks on AWS โ https://docs.databricks.com/aws/en/mlflow3/genai/eval-monitor/production-monitoring
[7] Billable usage system table reference | Databricks on AWS โ https://docs.databricks.com/aws/en/admin/system-tables/billing
2 weeks ago
Rate limits as a circuit breaker, loop alerts as the early warning, cost baseline for drift
I'd use all three, but for different jobs, and I wouldn't make the token cap the main signal.
1. Hard caps through AI Gateway. Keep them, but treat them as a circuit breaker, not a budget. AI Gateway rate limits are per minute (QPM/TPM) per endpoint, user, group or service principal, so a true daily cap isn't something you get natively. If each agent calls with its own service principal, you at least get per-agent limits on a shared endpoint. Set them high enough that a normal busy day never hits them.
2. Tool calls per request. This is the one that catches loops and retry storms early, and it's cheap: count tool spans per trace and alert above a fixed threshold, or when the same tool with the same arguments repeats. It needs no baseline, which is why it fires before any cost metric moves.
3. Cost per completed task vs. a 7-day baseline. Useful for drift (a prompt got longer, the retriever returns more chunks, a model swap), but it lags: by the time it alerts, a loop has already run for a while. "Completed" is also the tricky part, since failed or abandoned tasks are often where the spend is, so track cost per attempt too.
On data sources:
โข system.serving.endpoint_usage has token counts per request and requester. If you pass a usage_context in the request (agent name, task id), you can group by agent without splitting endpoints.
โข Inference tables give you the payloads, great for debugging a spike, less so as a metrics source.
โข Neither knows what a "task" is. For per-task numbers I'd lean on MLflow Tracing: one trace per task, tagged with agent and task id, with tool-call counts and token usage coming from the spans.
What I'd skip: alerting on raw tokens per agent per day. It mostly flags the agents that are busy, not the ones that are broken.
Are your agents calling the endpoints with separate service principals, or through one shared identity? That decides how much you get for free from the system tables.