This isn't a config problem on your end, it's a support gap. On-behalf-of auth for the standalone Unity AI Gateway beta endpoint isn't supported yet when you're deploying through databricks.agents.deploy().

Specifically, "ai-gateway" isn't a valid value in UserAuthPolicy.api_scopes right now, which is exactly why deployment fails as soon as you add it. There's also no MLflow resource type for standalone AI Gateway yet, so there's no SystemAuthPolicy or UserAuthPolicy resource path you can point at it either. It's just not wired up on the agent auth side at the moment.

A couple things worth checking before you go further down this path:

If what you actually want is a model serving or Foundation Model endpoint that happens to have AI Gateway features turned on, you don't need any of this. Call it like a normal serving endpoint using the serving endpoint scope, the AI Gateway branding on top doesn't change how auth works there.

If you specifically need the standalone Unity AI Gateway product (the one in the left nav), your options today are limited. You can fall back to service-principal or system auth instead of OBO, but you'll lose per-user attribution on those calls. The other option is to move that piece of the workflow into a Databricks App instead, since Apps support a broader set of OAuth scopes and that's the documented pattern for this kind of thing right now, agents.deploy() just isn't.

So don't add ai-gateway to UserAuthPolicy.api_scopes, it won't work. If a serving endpoint gets you what you need, use that. Otherwise, service-principal auth or a Databricks App are your workarounds until OBO support lands for standalone AI Gateway.