cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Using Databricks Docker CLI image with Asset Bundles - Azure

JacekJacek
New Contributor III

I'm trying to deploy asset bundle from a CI/CD pipeline, I'd like to use the docker databricks CLI image for that, but I can't get it to authenticate. I'm using entra service principal for my deployments and we are using TeamCity as our CI/CD tool. The cli image does not come with az cli preinstalled, so that auth is not possible, my organization has disabled personal access tokens as well, so I'm trying to use oauth m2m, but can't get it to work. I keep getting an error:

08:22:45  08:22:45 INFO start pid=1 version=0.238.0 args="/app/databricks, bundle, validate, -t, prototype-dev, -p, DEFAULT, --log-level=debug"
08:22:45  08:22:45 DEBUG Found bundle root at /my-bundle (file /my-bundle/databricks.yml) pid=1
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=load
08:22:45  08:22:45 INFO Phase: load pid=1 mutator=load
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=load mutator=seq
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=load mutator=seq mutator=EntryPoint
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=load mutator=seq mutator=scripts.preinit
08:22:45  08:22:45 DEBUG No script defined for preinit, skipping pid=1 mutator=load mutator=seq mutator=scripts.preinit
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=load mutator=seq mutator=ProcessRootIncludes
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq mutator=ProcessInclude(bundle/targets/prototype-dde.yml)
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq mutator=ProcessInclude(bundle/targets/prototype-uat.yml)
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq mutator=ProcessInclude(bundle/targets/prototype-prd.yml)
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq mutator=ProcessInclude(bundle/targets/prototype-dev.yml)
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq mutator=ProcessInclude(bundle/resources/source_types/file/pipelines/file_ingestion_pipeline.yml)
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq mutator=ProcessInclude(bundle/resources/source_types/file/jobs/configure_file_sources.yml)
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq mutator=ProcessInclude(bundle/resources/source_types/file/jobs/file_ingestion_job.yml)
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=load mutator=seq mutator=ProcessRootIncludes mutator=seq mutator=ProcessInclude(bundle/resources/source_types/file/variables.common.yml)
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=load mutator=seq mutator=VerifyCliVersion
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=load mutator=seq mutator=EnvironmentsToTargets
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=load mutator=seq mutator=ComputeIdToClusterId
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=load mutator=seq mutator=InitializeVariables
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=load mutator=seq mutator=DefineDefaultTarget(default)
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=load mutator=seq mutator=PythonMutator(load)
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=load mutator=seq mutator=validate:unique_resource_keys
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=load mutator=seq mutator=SelectTarget(prototype-dev)
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=<func>
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=<func>
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=initialize
08:22:45  08:22:45 INFO Phase: initialize pid=1 mutator=initialize
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=initialize mutator=seq
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=initialize mutator=seq mutator=validate:AllResourcesHaveValues
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=initialize mutator=seq mutator=RewriteSyncPaths
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=initialize mutator=seq mutator=SyncDefaultPath
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=initialize mutator=seq mutator=SyncInferRoot
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=initialize mutator=seq mutator=MergeJobClusters
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=initialize mutator=seq mutator=MergeJobParameters
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=initialize mutator=seq mutator=MergeJobTasks
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=initialize mutator=seq mutator=MergePipelineClusters
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=initialize mutator=seq mutator=InitializeWorkspaceClient
08:22:45  08:22:45 DEBUG Loading DEFAULT profile from /root/.databrickscfg pid=1 sdk=true
08:22:45  08:22:45 DEBUG Apply pid=1 mutator=initialize mutator=seq mutator=PopulateCurrentUser
08:22:45  08:22:45 DEBUG Loading DEFAULT profile from /root/.databrickscfg pid=1 sdk=true
08:22:45  08:22:45 INFO Ignoring pat auth, because databricks-cli is preferred pid=1 sdk=true
08:22:45  08:22:45 INFO Ignoring basic auth, because databricks-cli is preferred pid=1 sdk=true
08:22:45  08:22:45 INFO Ignoring oauth-m2m auth, because databricks-cli is preferred pid=1 sdk=true
08:22:45  08:22:45 DEBUG Running command: /app/databricks auth token --host https://adb-2355869874698299.19.azuredatabricks.net pid=1 sdk=true
08:22:45  Error: failed during request visitor: default auth: cannot configure default credentials, please check https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication to configure credentials for your preferred authentication method. Config: host=https://adb-xxx.azuredatabricks.net, profile=DEFAULT, azure_client_secret=***, azure_client_id=xxx, azure_tenant_id=xxx, client_id=xxx, client_secret=***, databricks_cli_path=/app/databricks. Env: ARM_CLIENT_SECRET, ARM_CLIENT_ID, ARM_TENANT_ID, DATABRICKS_CLI_PATH
08:22:45 
08:22:45  Name: xxx
08:22:45  Target: prototype-dev
08:22:45  Workspace:
08:22:45  Host: https://adb-xxx.azuredatabricks.net/
08:22:45 
08:22:45  Found 1 error

I'm invoking the cli this way:

docker run \
-v %teamcity.build.checkoutDir%:/my-bundle \
-v %teamcity.build.checkoutDir%/.databrickscfg:/root/.databrickscfg \
-w /my-bundle \
-e NO_PROXY=$NO_PROXY \
-e ARM_CLIENT_SECRET="%env.TF_VAR_client_secret%" \
-e ARM_CLIENT_ID="%env.TF_VAR_client_id%" \
-e ARM_TENANT_ID="%env.TF_VAR_tenant_id%" \
-e DATABRICKS_AUTH_TYPE="oauth-m2m" \
%docker.repository%/databricks/cli:0.238.0 bundle validate -t prototype-dev -p DEFAULT --log-level=debug

The .databrickscfg contents:

[DEFAULT]
host = https://adb-xxx.azuredatabricks.net/
client_id = %env.TF_VAR_client_id%
client_secret = %databricks.token%
auth_type = oauth-m2m

 Tried numerous combinations of config/env but nothing seems to work.. do I need a custom image with both databricks cli and az cli?

 

1 ACCEPTED SOLUTION

Accepted Solutions

JacekJacek
New Contributor III

OK, tested and now everything is working - according to the docs bundle settings are of highest priority, https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/authentication#auth-eval

No wonder none of my env vars or .databrickscfg settings worked. Oh well..

View solution in original post

5 REPLIES 5

JacekJacek
New Contributor III

The above is the output of bundle validate command, so before we even run deployment, but that would fail the same way ofc.

aj_dbx_
New Contributor II

MariuszK
Contributor III

Hi,

I think you should use these environment variables:DATABRICKS_CLIENT_ID, DATABRICKS_CLIENT_SECRET, DATABRICKS_ACCOUNT_ID, DATABRICKS_HOST

 

JacekJacek
New Contributor III

account_id is for authenticating to the accounts.azuredatabricks.net - I'm deploying a bundle to the workspace, but I just found that there was auth_type added in the bundle and that trumped all other methods / settings (strange), so now I've removed it and will give it another go...

JacekJacek
New Contributor III

OK, tested and now everything is working - according to the docs bundle settings are of highest priority, https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/authentication#auth-eval

No wonder none of my env vars or .databrickscfg settings worked. Oh well..

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group