Can Databricks federation policy support cross-cloud authentication?

Direo
Contributor II

Hi!

I'm exploring options for workload identity federation and have a question about cross-cloud scenarios.

Current Setup:

  • Azure Databricks workspace
  • Workloads running in GCP (planning to use GKE/Kubernetes)
  • Need to authenticate GCP-based workloads to Azure Databricks APIs without managing secrets

Question: Is the Databricks federation policy designed to support cross-cloud federation? Specifically, can I configure a service principal federation policy in Azure Databricks to accept tokens from a GCP Kubernetes cluster?

Looking at the documentation, I see Kubernetes is listed as a supported identity provider with this example configuration:
Issuer: https://kubernetes.default.svc
Audience: https://kubernetes.default.svc
Subject: system:serviceaccount:namespace:podname

My specific concerns:

  1. Would this work with a GKE cluster's external issuer URL instead of the internal kubernetes.default.svc?
  2. Are there any known limitations or considerations for cross-cloud federation scenarios?
  3. Has anyone successfully implemented GCP workload identity → Azure Databricks authentication?

Alternative considered: I'm aware I could potentially use Azure Entra ID as an intermediary, but I'm hoping to establish direct federation if possible to reduce complexity.

Any insights or experiences with cross-cloud federation would be greatly appreciated!

Thanks!