Can Databricks federation policy support cross-cloud authentication?

[Direo]

Hi!

I'm exploring options for workload identity federation and have a question about cross-cloud scenarios.

Current Setup:

• Azure Databricks workspace
• Workloads running in GCP (planning to use GKE/Kubernetes)
• Need to authenticate GCP-based workloads to Azure Databricks APIs without managing secrets

Question: Is the Databricks federation policy designed to support cross-cloud federation? Specifically, can I configure a service principal federation policy in Azure Databricks to accept tokens from a GCP Kubernetes cluster?

Looking at the documentation, I see Kubernetes is listed as a supported identity provider with this example configuration:
Issuer: https://kubernetes.default.svc
Audience: https://kubernetes.default.svc
Subject: system:serviceaccount:namespace:podname

My specific concerns:

1. Would this work with a GKE cluster's external issuer URL instead of the internal kubernetes.default.svc?
2. Are there any known limitations or considerations for cross-cloud federation scenarios?
3. Has anyone successfully implemented GCP workload identity → Azure Databricks authentication?

Alternative considered: I'm aware I could potentially use Azure Entra ID as an intermediary, but I'm hoping to establish direct federation if possible to reduce complexity.

Any insights or experiences with cross-cloud federation would be greatly appreciated!

Thanks!

[mark_ott]

Yes, Databricks federation policy can support cross-cloud authentication, allowing the use of external identity providers (IdPs) that may reside in different clouds. This includes scenarios where tokens issued by trusted IdPs—such as those for service principals running in different cloud platforms, like Azure, AWS, or GCP—can be federated for Databricks API access or Delta Sharing.

How Federation Policy Enables Cross-Cloud Authentication

• Databricks supports account-wide token federation and workload identity federation, which allow the configuration of federation policies that define trusted issuers (IdPs), including Kubernetes clusters and other cloud-native identity services.

• The platform validates tokens issued by these IdPs by referencing their well-known endpoints and JSON Web Key Sets (JWKS), provided the IdP is controlled and trusted by the organization.

• This mechanism allows, for example, a service principal in Azure Databricks to authenticate using tokens received from a GCP (Google Cloud Platform) Kubernetes cluster, as long as the federation policy is configured to accept that particular Kubernetes issuer as a trusted IdP