Hey sparkplug - Thanks for reaching out! 

To enable a service account in AKS to authenticate to Databricks using workload identity federation, you must create a service principal federation policy in Databricks that allows tokens issued by the Kubernetes cluster acting as the OIDC provider.

Federation Policy Example (for Kubernetes Service Account):

Key parameters:

• Issuer: OIDC endpoint of the Kubernetes cluster (typically https://kubernetes.default.svc for in-cluster workloads)
• Audience: Also usually https://kubernetes.default.svc
• Subject: The Kubernetes service account in the form system:serviceaccount:<namespace>:<serviceaccount-name>

Sample Policy (using Databricks CLI):

databricks account service-principal-federation-policy create <SERVICE_PRINCIPAL_NUMERIC_ID> --json \
'{
  "oidc_policy": {
    "issuer": "https://kubernetes.default.svc",
    "audiences": ["https://kubernetes.default.svc"],
    "subject": "system:serviceaccount:my-namespace:my-serviceaccount"
  }
}'

Corresponding JWT Claims:

{
  "iss": "https://kubernetes.default.svc",
  "aud": ["https://kubernetes.default.svc"],
  "sub": "system:serviceaccount:my-namespace:my-serviceaccount"
}

• You can also configure this via Databricks REST API or UI ("Federation policies" tab under service principal details). https://learn.microsoft.com/en-us/azure/databricks/dev-tools/auth/oauth-federation-policy

Terraform Example (resource arguments): https://github.com/databricks/terraform-provider-databricks/blob/main/docs/resources/service_princip...

• issuer (string): OIDC issuer URL
• audiences (list): List of valid audience values
• subject (string): Required Kubernetes service account identifier
• subject_claim (optional): Defaults to sub

Databricks Integration: Token Handling in AKS Pods

Databricks does not directly inject Databricks tokens into pods in AKS clusters.

How does the authentication work?

• With Azure Workload Identity Federation, the Kubernetes pod receives a short-lived, projected service account token.
• This token is used by the pod to exchange with Azure Entra ID (previously Azure AD) for an access token, which can then be used to access Databricks (or any other Azure resource).
• The workflow for a pod:
  • Receives a service account token via a projected volume.
  • The application in the pod uses this token (following the federation policy described above) to exchange for a Databricks OAuth token.
  • The Databricks CLI/SDK (or your code) handles the token exchange process using environment variables or token file paths provided by the Azure Workload Identity webhook.
    3 sources

YAML Pod Configuration Example (key extracts):

spec:
  serviceAccountName: my-serviceaccount
  volumes:
    - name: azure-projected-service-account-token
      projected:
        sources:
          - serviceAccountToken:
              audience: api://AzureADTokenExchange
              expirationSeconds: 3600
              path: azure-projected-service-account-token
  containers:
    - name: my-container
      volumeMounts:
        - mountPath: /databricks/secrets/azure-projected-service-account-token
          name: azure-projected-service-account-token
          readOnly: true
      env:
        - name: AZURE_CLIENT_ID
          value: <client-id>
        - name: AZURE_TENANT_ID
          value: <tenant>
        - name: AZURE_AUTHORITY_HOST
          value: "https://login.microsoftonline.com/"
        - name: AZURE_FEDERATED_TOKEN_FILE
          value: "/databricks/secrets/azure-projected-service-account-token/azure-projected-service-account-token"

• The Databricks SDK/CLI will look for AZURE_FEDERATED_TOKEN_FILE, perform the token exchange, and obtain a Databricks OAuth token.

Databricks Token Federation vs. OAuth m2m Flow

Workload Identity Federation (What you configure here)

• The workload (your application in a pod) obtains a federated (OIDC) token (from Kubernetes/Azure), then exchanges it for a short-lived Databricks OAuth token via the Databricks federation endpoint.
• No long-lived client secrets or tokens are managed or injected by Databricks.
• This model is more secure and easier to operate at scale (automatic rotation, least privilege, reduced secret sprawl). https://docs.databricks.com/aws/en/dev-tools/auth/oauth-federation

OAuth m2m (Machine-to-Machine) Flow

• Typically, the service (machine) uses a client_id and client_secret pre-shared with Databricks to obtain tokens.
• Secrets must be managed, rotated, and injected into workloads (increasing risk if leaked).
• Not as dynamic or tightly scoped as workload identity federation.