databricks sdk for python authentication failing

Sujith_i — Wed, 05 Feb 2025 17:04:08 GMT

I am trying to use databricks sdk for python to do some account level operations like creating groups and created a databricks config file locally n provided the profile name as argument to AccountClient but authentication keeps failing. the same config file is working fine with CLI so any insights on how to provide auth details for accountclient method:

My current code is like this:

from databricks.sdk import AccountClient
a = AccountClient(profile="account")

my databirckscfg has below details:
[account]
host = https://accounts.databricks.net
account_id = <>
client_id = <>
client_secret=<>

Re: databricks sdk for python authentication failing

mark_ott — Fri, 31 Oct 2025 15:09:41 GMT

Authentication for account-level operations with Databricks SDK for Python requires more than just referencing the profile name in your local .databrickscfg file. While the CLI consults .databrickscfg for profiles and can use them directly, the SDK's AccountClient often does not automatically read the config file for account-level authentication the way CLI does.

Correct Authentication Approach

You need to provide explicit authentication parameters when initializing AccountClient, as the SDK does not natively pick up .databrickscfg profiles for account-level OAuth or service principal flows. The recommended approach is:

python

from databricks.sdk import AccountClient

a = AccountClient(
    host="https://accounts.databricks.net",
    account_id="<account_id>",
    client_id="<client_id>",
    client_secret="<client_secret>"
)

Here, all necessary details from your .databrickscfg profile are provided as keyword arguments directly to AccountClient. This step is crucial for account-level endpoints because notebook-native/workspace authentication doesn't work for account-wide operations; and the SDK doesn't read account-level config profiles by default.

Why CLI Works but SDK Fails

The CLI searches for authentication information in environment variables or in .databrickscfg, but the Python SDK's account-level client expects details to be explicitly supplied, especially for OAuth/service principal flows. This discrepancy stems from CLI having a more robust config profile discovery than the SDK for account-level resources.

Additional Tips

Ensure that the host matches your cloud provider (e.g., AWS, Azure, GCP—URLs differ).
Double-check you have account admin privileges for these operations.
The config profile method works for workspace-level clients, but not for account clients.

If you want consistent behavior for SDK and CLI, you can wrap profile extraction yourself in Python, load values from .databrickscfg, and pass them explicitly when constructing AccountClient.

References