<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:taxo="http://purl.org/rss/1.0/modules/taxonomy/" version="2.0">
  <channel>
    <title>topic Per-agent guardrails: what do you alert on, tokens, retries, or cost per task? in Generative AI</title>
    <link>https://community.databricks.com/t5/generative-ai/per-agent-guardrails-what-do-you-alert-on-tokens-retries-or-cost/m-p/173743#M2216</link>
    <description>&lt;P&gt;I'm trying to work out what a sensible guardrail looks like for each agent when several of them share a few serving endpoints.&lt;/P&gt;&lt;P&gt;A plain token budget feels too blunt. An agent that retries a failing tool four times looks like a heavy user, while a cheap agent stuck in a loop can burn far more than its neighbours before anyone notices.&lt;/P&gt;&lt;P&gt;The options I'm weighing:&lt;/P&gt;&lt;P&gt;1. A hard daily cap per agent on tokens, through AI Gateway rate limits.&lt;/P&gt;&lt;P&gt;2. An alert on tool invocations per request, to catch loops and retry storms early.&lt;/P&gt;&lt;P&gt;3. An alert on cost per completed task against that agent's own 7-day baseline.&lt;/P&gt;&lt;P&gt;For those running agents in production on Databricks: which of these do you actually use, and which one did you regret? And is anything in the system tables or inference tables good enough to build that baseline from, or do you end up logging your own per-task numbers?&lt;/P&gt;&lt;P&gt;Happy to hear partial answers too, even just what you stopped doing.&lt;/P&gt;</description>
    <pubDate>Sun, 18 Oct 2026 00:56:24 GMT</pubDate>
    <dc:creator>DoTA</dc:creator>
    <dc:date>2026-10-18T00:56:24Z</dc:date>
    <item>
      <title>Per-agent guardrails: what do you alert on, tokens, retries, or cost per task?</title>
      <link>https://community.databricks.com/t5/generative-ai/per-agent-guardrails-what-do-you-alert-on-tokens-retries-or-cost/m-p/173743#M2216</link>
      <description>&lt;P&gt;I'm trying to work out what a sensible guardrail looks like for each agent when several of them share a few serving endpoints.&lt;/P&gt;&lt;P&gt;A plain token budget feels too blunt. An agent that retries a failing tool four times looks like a heavy user, while a cheap agent stuck in a loop can burn far more than its neighbours before anyone notices.&lt;/P&gt;&lt;P&gt;The options I'm weighing:&lt;/P&gt;&lt;P&gt;1. A hard daily cap per agent on tokens, through AI Gateway rate limits.&lt;/P&gt;&lt;P&gt;2. An alert on tool invocations per request, to catch loops and retry storms early.&lt;/P&gt;&lt;P&gt;3. An alert on cost per completed task against that agent's own 7-day baseline.&lt;/P&gt;&lt;P&gt;For those running agents in production on Databricks: which of these do you actually use, and which one did you regret? And is anything in the system tables or inference tables good enough to build that baseline from, or do you end up logging your own per-task numbers?&lt;/P&gt;&lt;P&gt;Happy to hear partial answers too, even just what you stopped doing.&lt;/P&gt;</description>
      <pubDate>Sun, 18 Oct 2026 00:56:24 GMT</pubDate>
      <guid>https://community.databricks.com/t5/generative-ai/per-agent-guardrails-what-do-you-alert-on-tokens-retries-or-cost/m-p/173743#M2216</guid>
      <dc:creator>DoTA</dc:creator>
      <dc:date>2026-10-18T00:56:24Z</dc:date>
    </item>
    <item>
      <title>Re: Per-agent guardrails: what do you alert on, tokens, retries, or cost per task?</title>
      <link>https://community.databricks.com/t5/generative-ai/per-agent-guardrails-what-do-you-alert-on-tokens-retries-or-cost/m-p/173844#M2219</link>
      <description>&lt;P&gt;Hi&amp;nbsp;&lt;a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/125801"&gt;@DoTA&lt;/a&gt;,&lt;/P&gt;
&lt;P&gt;Good question.&amp;nbsp;I'd actually use all three of your options, but each one does a different job, and the first needs a small change.&lt;BR /&gt;&lt;BR /&gt;&lt;/P&gt;
&lt;DIV class="du-bois-light-typography css-mdu5st" data-genai-markdown-block="true"&gt;On the hard daily cap,&amp;nbsp;Unity Gateway doesn't do daily caps. Its rate limits are set as queries per minute (QPM) and tokens per minute (TPM), at the service level, as a default per user, or as custom limits for individual users, service principals and groups. See&amp;nbsp;&lt;A href="https://learn.microsoft.com/en-us/azure/databricks/ai-gateway/rate-limits" target="_blank"&gt;this&lt;/A&gt;&amp;nbsp;for reference.&amp;nbsp;&amp;nbsp;A per-minute TPM limit is still worth having, but think of it as a safety ceiling that stops one agent from starving its neighbours on a shared endpoint. It won't keep spending down. Two things make it work much better...&lt;/DIV&gt;
&lt;OL class="css-0"&gt;
&lt;LI&gt;Run each agent as its own service principal. Limits and usage records are keyed to who makes the call, so agents sharing a single identity can't be distinguished.&lt;/LI&gt;
&lt;LI&gt;Set the ceiling well above normal peak traffic. If it's set tight, a legitimate burst gets 429 errors, your retry logic kicks in, and you've made the retry storm you were trying to prevent.&lt;/LI&gt;
&lt;/OL&gt;
&lt;DIV class="du-bois-light-typography css-mdu5st" data-genai-markdown-block="true"&gt;Also note that TPM limits can't be applied to endpoints that serve custom models or agents themselves. They apply only to the model endpoints that agents call. See&lt;SPAN&gt;&amp;nbsp;&lt;A href="https://learn.microsoft.com/en-us/azure/databricks/ai-gateway/configure-ai-gateway-endpoints" target="_blank"&gt;this&lt;/A&gt; for reference.&lt;/SPAN&gt;&lt;/DIV&gt;
&lt;DIV class="du-bois-light-typography css-mdu5st" data-genai-markdown-block="true"&gt;&amp;nbsp;&lt;/DIV&gt;
&lt;DIV class="du-bois-light-typography css-mdu5st" data-genai-markdown-block="true"&gt;On tool calls per request, this is the best early warning of the three, because loops and retry storms show up in tool-call counts well before they show up in the bill. If your tools sit behind MCP services in the gateway, you can put a QPM limit on those too, which gives you a hard stop on runaway tool calls. Token limits don't apply to MCP services.&lt;/DIV&gt;
&lt;DIV class="du-bois-light-typography css-mdu5st" data-genai-markdown-block="true"&gt;&amp;nbsp;&lt;/DIV&gt;
&lt;DIV class="du-bois-light-typography css-mdu5st" data-genai-markdown-block="true"&gt;On cost per task against a 7-day baseline, this is the signal you're really after, but it depends on how you define a "task", and the gateway can't see that. Usage data is recorded per request. The&amp;nbsp; newer&amp;nbsp; &amp;nbsp;&lt;SPAN class="du-bois-light-typography css-1ifq98"&gt;&lt;CODE&gt;system.ai_gateway.usage&lt;/CODE&gt;&lt;/SPAN&gt;&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;table has a&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;&lt;SPAN class="du-bois-light-typography css-1ifq98"&gt;&lt;CODE&gt;request_id&lt;/CODE&gt;&lt;/SPAN&gt;&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;plus an&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;&lt;SPAN class="du-bois-light-typography css-1ifq98"&gt;&lt;CODE&gt;invocation_id&lt;/CODE&gt;&lt;/SPAN&gt;&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;for each model call, so multi-turn agent calls can be grouped together (&lt;A href="https://learn.microsoft.com/en-us/azure/databricks/ai-gateway/usage-tracking" target="_blank"&gt;Reference&lt;/A&gt;).&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;The older&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;&lt;SPAN class="du-bois-light-typography css-1ifq98"&gt;&lt;CODE&gt;system.serving.endpoint_usage&lt;/CODE&gt;&lt;/SPAN&gt;&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;table has a&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;&lt;SPAN class="du-bois-light-typography css-1ifq98"&gt;&lt;CODE&gt;usage_context&lt;/CODE&gt;&lt;/SPAN&gt;&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;map where you can pass your own identifiers, such as an agent name or task ID. These tables are good for showing who is using how much, but they don't know where a task begins or ends, or whether it finished.&lt;/DIV&gt;
&lt;DIV class="du-bois-light-typography css-mdu5st" data-genai-markdown-block="true"&gt;&amp;nbsp;&lt;/DIV&gt;
&lt;DIV class="du-bois-light-typography css-mdu5st" data-genai-markdown-block="true"&gt;For the per-task baseline, the best source is MLflow tracing. Each trace records input and output tokens and estimated cost for every LLM call, rolled up to the trace level (&lt;A href="https://mlflow.org/docs/latest/genai/tracing/token-usage-cost/index.html" target="_blank"&gt;Reference&lt;/A&gt;).&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;If one trace equals one task, you get tool-span counts, token cost and outcome in the same place. Tag each trace with the agent name and a completed or failed status, and the baseline becomes a simple aggregation. You still have to add those tags yourself, but you don't need a separate logging pipeline.&lt;/DIV&gt;
&lt;DIV class="du-bois-light-typography css-mdu5st" data-genai-markdown-block="true"&gt;&amp;nbsp;&lt;/DIV&gt;
&lt;DIV class="du-bois-light-typography css-mdu5st" data-genai-markdown-block="true"&gt;As a starting point, give each agent its own service principal and pass the agent name in your request metadata. Then turn on tracing and build the tool-calls-per-trace alert, since it's the cheapest and catches problems earliest. Add the cost-per-task baseline once you have a week of tagged traces. Keep the per-principal TPM limit loose, as a safety net rather than a budget. If you can't use traces, the usage system table grouped by requester and&amp;nbsp;&lt;SPAN class="du-bois-light-typography css-1ifq98"&gt;&lt;CODE&gt;usage_context&lt;/CODE&gt;&lt;/SPAN&gt;&amp;nbsp;is a reasonable fallback, but it gives you cost per request, not cost per task.&lt;/DIV&gt;
&lt;DIV class="du-bois-light-typography css-mdu5st" data-genai-markdown-block="true"&gt;&amp;nbsp;&lt;/DIV&gt;
&lt;DIV class="du-bois-light-typography css-mdu5st" data-genai-markdown-block="true"&gt;
&lt;P class="p1"&gt;&lt;FONT size="2" color="#FF6600"&gt;&lt;STRONG&gt;&lt;I&gt;If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.&lt;/I&gt;&lt;/STRONG&gt;&lt;/FONT&gt;&lt;I&gt;&lt;/I&gt;&lt;/P&gt;
&lt;/DIV&gt;</description>
      <pubDate>Mon, 19 Oct 2026 11:11:48 GMT</pubDate>
      <guid>https://community.databricks.com/t5/generative-ai/per-agent-guardrails-what-do-you-alert-on-tokens-retries-or-cost/m-p/173844#M2219</guid>
      <dc:creator>Ashwin_DSA</dc:creator>
      <dc:date>2026-10-19T11:11:48Z</dc:date>
    </item>
    <item>
      <title>Re: Per-agent guardrails: what do you alert on, tokens, retries, or cost per task?</title>
      <link>https://community.databricks.com/t5/generative-ai/per-agent-guardrails-what-do-you-alert-on-tokens-retries-or-cost/m-p/173908#M2222</link>
      <description>&lt;P class=""&gt;&lt;STRONG&gt;Short answer:&lt;/STRONG&gt;&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;I'd use all three, but at different layers. A per-agent rate limit is the hard backstop, a calls-per-request alert is the early warning for loops and retry storms, and cost per completed task against a 7-day baseline is a review metric, not a guardrail. What makes all three work is giving every agent its own identity (a service principal) and tagging every request with the agent and task, because both the rate limiter and&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;system.ai_gateway.usage&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;key on who called and which tags came with the call [1][2].&lt;/P&gt;&lt;H2&gt;1. Hard cap: per-agent rate limit, sized per hour&lt;/H2&gt;&lt;P class=""&gt;Unity Gateway rate limits are set per minute or per hour, on requests and tokens, at four scopes: the whole service, a default per user, a specific user or service principal, and a group [1][3]. There is no daily window, so a "daily cap" becomes an hourly token ceiling sized at a few times the agent's normal peak.&lt;/P&gt;&lt;OL class=""&gt;&lt;LI&gt;Run each agent as its own service principal and give it a custom limit. User and service principal limits take priority over group limits [1].&lt;/LI&gt;&lt;LI&gt;Keep a service-level limit as well. It is a global ceiling for everyone sharing the endpoint [1].&lt;/LI&gt;&lt;LI&gt;Make the agent stop on HTTP 429 after a bounded backoff instead of retrying forever.&lt;/LI&gt;&lt;/OL&gt;&lt;P class=""&gt;The relevant part of the model service config looks like this [3]:&lt;/P&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&amp;nbsp;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;"&lt;/SPAN&gt;&lt;SPAN&gt;rate_limits&lt;/SPAN&gt;&lt;SPAN&gt;"&lt;/SPAN&gt;&lt;SPAN&gt;: &lt;/SPAN&gt;&lt;SPAN&gt;[&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;{&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;"&lt;/SPAN&gt;&lt;SPAN&gt;key&lt;/SPAN&gt;&lt;SPAN&gt;"&lt;/SPAN&gt;&lt;SPAN&gt;:&lt;/SPAN&gt; &lt;SPAN&gt;"&lt;/SPAN&gt;&lt;SPAN&gt;RATE_LIMIT_KEY_SERVICE_PRINCIPAL&lt;/SPAN&gt;&lt;SPAN&gt;"&lt;/SPAN&gt;&lt;SPAN&gt;,&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;"&lt;/SPAN&gt;&lt;SPAN&gt;principal&lt;/SPAN&gt;&lt;SPAN&gt;"&lt;/SPAN&gt;&lt;SPAN&gt;:&lt;/SPAN&gt; &lt;SPAN&gt;"&lt;/SPAN&gt;&lt;SPAN&gt;&amp;lt;agent-sp-application-id&amp;gt;&lt;/SPAN&gt;&lt;SPAN&gt;"&lt;/SPAN&gt;&lt;SPAN&gt;,&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;"&lt;/SPAN&gt;&lt;SPAN&gt;renewal_period&lt;/SPAN&gt;&lt;SPAN&gt;"&lt;/SPAN&gt;&lt;SPAN&gt;:&lt;/SPAN&gt; &lt;SPAN&gt;"&lt;/SPAN&gt;&lt;SPAN&gt;RATE_LIMIT_RENEWAL_PERIOD_HOUR&lt;/SPAN&gt;&lt;SPAN&gt;"&lt;/SPAN&gt;&lt;SPAN&gt;,&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;"&lt;/SPAN&gt;&lt;SPAN&gt;tokens&lt;/SPAN&gt;&lt;SPAN&gt;"&lt;/SPAN&gt;&lt;SPAN&gt;:&lt;/SPAN&gt; &lt;SPAN&gt;2000000&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;}&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;]&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;P class=""&gt;Two gotchas. The limiter records usage after a response is sent, so concurrent bursts can briefly overshoot; treat it as a backstop, not an exact budget [1]. And if you are still on AI Gateway on serving endpoints (legacy), TPM limits can't be applied to endpoints that serve custom models or agents, so put the token limit on the foundation model endpoint the agent calls [4].&lt;/P&gt;&lt;H2&gt;2. Early warning: model calls and tool calls per request&lt;/H2&gt;&lt;P class=""&gt;This is the one that catches the cheap agent stuck in a loop. In&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;system.ai_gateway.usage, every inference call gets its own&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;invocation_id, and multi-turn agent calls share a&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;request_id&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;[2]. If your tools are exposed as Unity Gateway MCP services,&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;mcp_metadata.tool_name&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;is logged too. That gives you calls per request straight from the table:&lt;/P&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&amp;nbsp;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;WITH&lt;/SPAN&gt;&lt;SPAN&gt; per_request &lt;/SPAN&gt;&lt;SPAN&gt;AS&lt;/SPAN&gt;&lt;SPAN&gt; (&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;SELECT&lt;/SPAN&gt;&lt;SPAN&gt; request_tags['agent'] &lt;/SPAN&gt;&lt;SPAN&gt;AS&lt;/SPAN&gt;&lt;SPAN&gt; agent,&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;request_id,&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;count&lt;/SPAN&gt;&lt;SPAN&gt;(&lt;/SPAN&gt;&lt;SPAN&gt;DISTINCT&lt;/SPAN&gt; &lt;SPAN&gt;CASE&lt;/SPAN&gt; &lt;SPAN&gt;WHEN&lt;/SPAN&gt;&lt;SPAN&gt; service_type &lt;/SPAN&gt;&lt;SPAN&gt;=&lt;/SPAN&gt; &lt;SPAN&gt;'&lt;/SPAN&gt;&lt;SPAN&gt;MODEL_SERVICE&lt;/SPAN&gt;&lt;SPAN&gt;'&lt;/SPAN&gt; &lt;SPAN&gt;THEN&lt;/SPAN&gt;&lt;SPAN&gt; invocation_id &lt;/SPAN&gt;&lt;SPAN&gt;END&lt;/SPAN&gt;&lt;SPAN&gt;) &lt;/SPAN&gt;&lt;SPAN&gt;AS&lt;/SPAN&gt;&lt;SPAN&gt; model_calls,&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;count&lt;/SPAN&gt;&lt;SPAN&gt;(&lt;/SPAN&gt;&lt;SPAN&gt;CASE&lt;/SPAN&gt; &lt;SPAN&gt;WHEN&lt;/SPAN&gt;&lt;SPAN&gt; service_type &lt;/SPAN&gt;&lt;SPAN&gt;=&lt;/SPAN&gt; &lt;SPAN&gt;'&lt;/SPAN&gt;&lt;SPAN&gt;MCP_SERVICE&lt;/SPAN&gt;&lt;SPAN&gt;'&lt;/SPAN&gt; &lt;SPAN&gt;THEN&lt;/SPAN&gt; &lt;SPAN&gt;mcp_metadata&lt;/SPAN&gt;&lt;SPAN&gt;.&lt;/SPAN&gt;&lt;SPAN&gt;tool_name&lt;/SPAN&gt; &lt;SPAN&gt;END&lt;/SPAN&gt;&lt;SPAN&gt;) &lt;/SPAN&gt;&lt;SPAN&gt;AS&lt;/SPAN&gt;&lt;SPAN&gt; tool_calls,&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;sum&lt;/SPAN&gt;&lt;SPAN&gt;(total_tokens) &lt;/SPAN&gt;&lt;SPAN&gt;AS&lt;/SPAN&gt;&lt;SPAN&gt; tokens&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;FROM&lt;/SPAN&gt; &lt;SPAN&gt;system&lt;/SPAN&gt;&lt;SPAN&gt;.&lt;/SPAN&gt;&lt;SPAN&gt;ai_gateway&lt;/SPAN&gt;&lt;SPAN&gt;.usage&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;WHERE&lt;/SPAN&gt;&lt;SPAN&gt; event_time &lt;/SPAN&gt;&lt;SPAN&gt;&amp;gt;=&lt;/SPAN&gt; &lt;SPAN&gt;current_timestamp&lt;/SPAN&gt;&lt;SPAN&gt;() &lt;/SPAN&gt;&lt;SPAN&gt;-&lt;/SPAN&gt;&lt;SPAN&gt; INTERVAL &lt;/SPAN&gt;&lt;SPAN&gt;8&lt;/SPAN&gt; &lt;SPAN&gt;DAYS&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;GROUP BY&lt;/SPAN&gt;&lt;SPAN&gt; ALL&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;)&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;SELECT&lt;/SPAN&gt;&lt;SPAN&gt; agent,&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;count&lt;/SPAN&gt;&lt;SPAN&gt;(&lt;/SPAN&gt;&lt;SPAN&gt;*&lt;/SPAN&gt;&lt;SPAN&gt;) &lt;/SPAN&gt;&lt;SPAN&gt;AS&lt;/SPAN&gt;&lt;SPAN&gt; requests,&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;percentile(model_calls, &lt;/SPAN&gt;&lt;SPAN&gt;0&lt;/SPAN&gt;&lt;SPAN&gt;.&lt;/SPAN&gt;&lt;SPAN&gt;95&lt;/SPAN&gt;&lt;SPAN&gt;) &lt;/SPAN&gt;&lt;SPAN&gt;AS&lt;/SPAN&gt;&lt;SPAN&gt; p95_model_calls,&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;max&lt;/SPAN&gt;&lt;SPAN&gt;(model_calls) &lt;/SPAN&gt;&lt;SPAN&gt;AS&lt;/SPAN&gt;&lt;SPAN&gt; max_model_calls,&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;max&lt;/SPAN&gt;&lt;SPAN&gt;(tool_calls) &lt;/SPAN&gt;&lt;SPAN&gt;AS&lt;/SPAN&gt;&lt;SPAN&gt; max_tool_calls,&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;percentile(tokens, &lt;/SPAN&gt;&lt;SPAN&gt;0&lt;/SPAN&gt;&lt;SPAN&gt;.&lt;/SPAN&gt;&lt;SPAN&gt;95&lt;/SPAN&gt;&lt;SPAN&gt;) &lt;/SPAN&gt;&lt;SPAN&gt;AS&lt;/SPAN&gt;&lt;SPAN&gt; p95_tokens&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;FROM&lt;/SPAN&gt;&lt;SPAN&gt; per_request&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&lt;SPAN&gt;GROUP BY&lt;/SPAN&gt;&lt;SPAN&gt; agent;&lt;/SPAN&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;DIV class=""&gt;&lt;DIV class=""&gt;&amp;nbsp;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;P class=""&gt;If your agent framework sends each turn as a separate HTTP request,&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;request_id&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;won't tie a task together. In that case send&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;agent&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;and&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;task_id&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;in the&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;Databricks-Ai-Gateway-Request-Tags&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;header and group by&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;request_tags['task_id']&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;instead [5]. Alert when&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;max_model_calls&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;goes above a multiple of that agent's own p95.&lt;/P&gt;&lt;P class=""&gt;The table is for alerting after the fact. To kill a loop while it runs, also enforce a max-steps or max-tool-calls limit inside the agent code.&lt;/P&gt;&lt;H2&gt;3. Cost per completed task against a 7-day baseline&lt;/H2&gt;&lt;P class=""&gt;Useful for spotting drift after a prompt or model change, but it fires after the money is spent. Two practical points:&lt;/P&gt;&lt;UL class=""&gt;&lt;LI&gt;&lt;STRONG&gt;"Completed" isn't in the gateway data.&lt;/STRONG&gt;&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;The gateway knows status codes, not whether the task succeeded. Log task outcome from the app, or use MLflow traces with production monitoring and join on your task id [6].&lt;/LI&gt;&lt;LI&gt;&lt;STRONG&gt;Use tokens for the daily view, billing for reconciliation.&lt;/STRONG&gt;&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;system.billing.usage&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;records are typically available within 12 hours, so it works for a 7-day baseline but not for same-day alerts [7].&lt;/LI&gt;&lt;/UL&gt;&lt;H2&gt;What I'd do&lt;/H2&gt;&lt;P class=""&gt;Skip the single shared daily token budget; that's exactly the noisy-neighbour problem you describe. Start with one service principal per agent and an hourly token limit on each, plus a max-steps limit in code. Add the calls-per-request alert from the query above, and review cost per completed task weekly rather than alerting on it.&lt;/P&gt;&lt;H2&gt;References&lt;/H2&gt;&lt;P class=""&gt;[1] Apply rate limits to model and MCP services | Databricks on AWS —&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;&lt;A class="" href="https://docs.databricks.com/aws/en/ai-gateway/rate-limits" target="_blank" rel="noopener noreferrer"&gt;https://docs.databricks.com/aws/en/ai-gateway/rate-limits&lt;/A&gt;&lt;SPAN&gt;&amp;nbsp;&lt;BR /&gt;&lt;/SPAN&gt;[2] Track model usage | Databricks on AWS —&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;&lt;A class="" href="https://docs.databricks.com/aws/en/ai-gateway/usage-tracking" target="_blank" rel="noopener noreferrer"&gt;https://docs.databricks.com/aws/en/ai-gateway/usage-tracking&lt;/A&gt;&lt;SPAN&gt;&amp;nbsp;&lt;BR /&gt;&lt;/SPAN&gt;[3] Model Service | Databricks API Reference —&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;&lt;A class="" href="https://docs.databricks.com/api/ai-gateway/v1/model-service" target="_blank" rel="noopener noreferrer"&gt;https://docs.databricks.com/api/ai-gateway/v1/model-service&lt;/A&gt;&lt;SPAN&gt;&amp;nbsp;&lt;BR /&gt;&lt;/SPAN&gt;[4] Configure governance on model serving endpoints (legacy) | Databricks on AWS —&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;&lt;A class="" href="https://docs.databricks.com/aws/en/ai-gateway/configure-ai-gateway-endpoints" target="_blank" rel="noopener noreferrer"&gt;https://docs.databricks.com/aws/en/ai-gateway/configure-ai-gateway-endpoints&lt;/A&gt;&lt;SPAN&gt;&amp;nbsp;&lt;BR /&gt;&lt;/SPAN&gt;[5] Request tagging | Databricks on AWS —&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;&lt;A class="" href="https://docs.databricks.com/aws/en/ai-gateway/request-tagging" target="_blank" rel="noopener noreferrer"&gt;https://docs.databricks.com/aws/en/ai-gateway/request-tagging&lt;/A&gt;&lt;SPAN&gt;&amp;nbsp;&lt;BR /&gt;&lt;/SPAN&gt;[6] Set up production monitoring | Databricks on AWS —&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;&lt;A class="" href="https://docs.databricks.com/aws/en/mlflow3/genai/eval-monitor/production-monitoring" target="_blank" rel="noopener noreferrer"&gt;https://docs.databricks.com/aws/en/mlflow3/genai/eval-monitor/production-monitoring&lt;/A&gt;&lt;SPAN&gt;&amp;nbsp;&lt;BR /&gt;&lt;/SPAN&gt;[7] Billable usage system table reference | Databricks on AWS —&lt;SPAN&gt;&amp;nbsp;&lt;/SPAN&gt;&lt;A class="" href="https://docs.databricks.com/aws/en/admin/system-tables/billing" target="_blank" rel="noopener noreferrer"&gt;https://docs.databricks.com/aws/en/admin/system-tables/billing&lt;/A&gt;&lt;/P&gt;</description>
      <pubDate>Mon, 19 Oct 2026 18:15:34 GMT</pubDate>
      <guid>https://community.databricks.com/t5/generative-ai/per-agent-guardrails-what-do-you-alert-on-tokens-retries-or-cost/m-p/173908#M2222</guid>
      <dc:creator>VladTamas</dc:creator>
      <dc:date>2026-10-19T18:15:34Z</dc:date>
    </item>
    <item>
      <title>Re: Per-agent guardrails: what do you alert on, tokens, retries, or cost per task?</title>
      <link>https://community.databricks.com/t5/generative-ai/per-agent-guardrails-what-do-you-alert-on-tokens-retries-or-cost/m-p/174900#M2245</link>
      <description>&lt;P&gt;Rate limits as a circuit breaker, loop alerts as the early warning, cost baseline for drift&lt;/P&gt;&lt;P&gt;I'd use all three, but for different jobs, and I wouldn't make the token cap the main signal.&lt;/P&gt;&lt;P&gt;1. Hard caps through AI Gateway. Keep them, but treat them as a circuit breaker, not a budget. AI Gateway rate limits are per minute (QPM/TPM) per endpoint, user, group or service principal, so a true daily cap isn't something you get natively. If each agent calls with its own service principal, you at least get per-agent limits on a shared endpoint. Set them high enough that a normal busy day never hits them.&lt;/P&gt;&lt;P&gt;2. Tool calls per request. This is the one that catches loops and retry storms early, and it's cheap: count tool spans per trace and alert above a fixed threshold, or when the same tool with the same arguments repeats. It needs no baseline, which is why it fires before any cost metric moves.&lt;/P&gt;&lt;P&gt;3. Cost per completed task vs. a 7-day baseline. Useful for drift (a prompt got longer, the retriever returns more chunks, a model swap), but it lags: by the time it alerts, a loop has already run for a while. "Completed" is also the tricky part, since failed or abandoned tasks are often where the spend is, so track cost per attempt too.&lt;/P&gt;&lt;P&gt;On data sources:&lt;/P&gt;&lt;P&gt;• system.serving.endpoint_usage has token counts per request and requester. If you pass a usage_context in the request (agent name, task id), you can group by agent without splitting endpoints.&lt;/P&gt;&lt;P&gt;• Inference tables give you the payloads, great for debugging a spike, less so as a metrics source.&lt;/P&gt;&lt;P&gt;• Neither knows what a "task" is. For per-task numbers I'd lean on MLflow Tracing: one trace per task, tagged with agent and task id, with tool-call counts and token usage coming from the spans.&lt;/P&gt;&lt;P&gt;What I'd skip: alerting on raw tokens per agent per day. It mostly flags the agents that are busy, not the ones that are broken.&lt;/P&gt;&lt;P&gt;Are your agents calling the endpoints with separate service principals, or through one shared identity? That decides how much you get for free from the system tables.&lt;/P&gt;</description>
      <pubDate>Thu, 29 Oct 2026 05:19:03 GMT</pubDate>
      <guid>https://community.databricks.com/t5/generative-ai/per-agent-guardrails-what-do-you-alert-on-tokens-retries-or-cost/m-p/174900#M2245</guid>
      <dc:creator>FrankAzzollini</dc:creator>
      <dc:date>2026-10-29T05:19:03Z</dc:date>
    </item>
  </channel>
</rss>

