Databricks CLI token creation fails with “cannot configure default credentials”

[gannicus]

Hello, I have been generating a Databricks personal access token in my YAML-based CI pipeline using a bash script. The pipeline installs the Databricks CLI and then creates a token using a Service Principal (Azure AD application) credentials.

Current working approach (previously working)

#!/bin/bash

dbx_host="${1}"
dbx_client_id="${2}"
dbx_client_secret="${3}"

# Set the Environment Variables for Databricks authentication
export DATABRICKS_HOST=$dbx_host
export DATABRICKS_CLIENT_ID=$dbx_client_id
export DATABRICKS_CLIENT_SECRET=$dbx_client_secret

echo "Creating a new Databricks token"

response=$(databricks tokens create \
  --lifetime-seconds 31536000 \
  --comment "Token for SPN for EDH Data Access. Validity 1 year.")

echo "Token Created Successfully"

token=$(echo $response | jq -r '.token_value')
token_id=$(echo $response | jq -r '.token_info.token_id')
expiry_time=$(echo $response | jq -r '.token_info.expiry_time')

This used to work fine for generating tokens.

Issue: Recently, the same pipeline started failing with the following error:

Error: default auth: cannot configure default credentials, please check https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication to configure credentials for your preferred authentication method.

Config: host=https://***, account_id=***, workspace_id=***, profile=DEFAULT, azure_tenant_id=***, client_id=***, client_secret=***

Env: DATABRICKS_HOST, DATABRICKS_CLIENT_ID, DATABRICKS_CLIENT_SECRET

The documentation link provided in the error message does not really help in identifying what exactly needs to be changed or how to fix this specific CI/CD use case.

Has there been a recent change in Databricks CLI authentication (especially unified authentication) that breaks Service Principal authentication using DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET environment variables?

Any guidance or migration steps would be appreciated.