Service principle used in Bitbucket CICD pipelines not working

[paulchen]

The Databricks Asset Bundle is used for the Bitbucket CICD pipelines.

The service principle is used in both of the local Databricks configuration and Bitbucket CICD environment.

The service principle is only working in the local environment for deployment, but not in Bitbucket CICD environment.

The SP config similar to :

[DEFAULT]
host = <host>
account_id = <account_id>
client_id = <client_id>
client_secret = <client_secret>

If I use personal user configuration in local and Bitbucket CICD environment, the DAB worked fine in both envs without any issue.

[sarahbhord]

Hey PaulChen

Having your Databricks service principal (SP) work locally but fail in Bitbucket CI/CD usually means environment variables aren’t set up right, or the pipeline is falling back to an unexpected config.

Quick checklist:

• Set all SP credentials (DATABRICKS_HOST, DATABRICKS_ACCOUNT_ID, DATABRICKS_CLIENT_ID, DATABRICKS_CLIENT_SECRET) as Bitbucket repo/project variables—don’t hard-code.
• Ensure the pipeline script actually exports those vars before running Databricks CLI/DAB.
• Make sure your pipeline is referencing the right profile/environment and not a user config by mistake.
• Confirm your service principal has all the permissions it needs in Databricks (deployments, workflow runs, etc.).
• Match the Databricks CLI version in Bitbucket to what you use locally.

If you’re referencing a .databrickscfg file in CI, build it from env vars during your pipeline for security. Check error messages—missing or mistyped env var names are common causes. If you stay stuck, echo a redacted config dump in CI to compare with local.

Let me know if this was helpful! 

Sarah