Databricks Docker CLI image - how to debug Terraform when deploying Asset Bundle

JacekJacek — Thu, 06 Feb 2025 07:45:24 GMT

We're having issues when deploying asset bundle using docker databricks cli image.

Validation part passes OK:

14:21:15 Name: test 14:21:15 Target: prototype-dev 14:21:15 Workspace: 14:21:15 Host: https://adb-xxx.azuredatabricks.net/ 14:21:15 User: *** 14:21:15 Path: /Workspace/Users/***/.bundle/test/prototype-dev 14:21:15 14:21:15 Validation OK!

And the bundle upload works OK as well:

14:21:20 Uploading bundle files to /Workspace/Users/***/.bundle/test/prototype-dev/files... 14:21:27 Deploying resources... 14:21:29 Updating deployment state... 14:21:30 Deployment complete!

But then when cli runs terraform it fails:

14:21:30 Error: terraform apply: exit status 1 14:21:30 14:21:30 Error: cannot create job: Unauthorized network access to workspace: xxx 14:21:30 14:21:30 with databricks_job.configure-file-sources, 14:21:30 on bundle.tf.json line 96, in resource.databricks_job.configure-file-sources: 14:21:30 96: }, 14:21:30 14:21:30 14:21:30 Error: cannot create pipeline: Unauthorized network access to workspace: xxx 14:21:30 14:21:30 with databricks_pipeline.file_ingestion_pipeline, 14:21:30 on bundle.tf.json line 313, in resource.databricks_pipeline.file_ingestion_pipeline: 14:21:30 313: } 14:21:30 14:21:30 Error: terraform apply: exit status 1 14:21:30 14:21:30 Error: cannot create job: Unauthorized network access to workspace: xxx 14:21:30 14:21:30 with databricks_job.configure-file-sources, 14:21:30 on bundle.tf.json line 96, in resource.databricks_job.configure-file-sources: 14:21:30 96: }, 14:21:30 14:21:30 14:21:30 Error: cannot create pipeline: Unauthorized network access to workspace: xxx 14:21:30 14:21:30 with databricks_pipeline.file_ingestion_pipeline, 14:21:30 on bundle.tf.json line 313, in resource.databricks_pipeline.file_ingestion_pipeline: 14:21:30 313: } 14:21:30

we are running the deploy command from TeamCity like this:

NO_PROXY="adb-xxx.azuredatabricks.net" docker run \ -v %teamcity.build.checkoutDir%:/my-bundle \ -v %teamcity.build.checkoutDir%/.databrickscfg:/root/.databrickscfg \ -w /my-bundle \ -e ARM_CLIENT_SECRET="%env.TF_VAR_client_secret%" \ -e ARM_CLIENT_ID="%env.TF_VAR_client_id%" \ -e ARM_TENANT_ID="%env.TF_VAR_tenant_id%" \ -e NO_PROXY=$NO_PROXY \ -e TF_LOG="DEBUG" \ %docker.repository%/databricks/cli:0.238.0 bundle deploy -t prototype-dev -p DEFAULT --debug

And we can see the debugging output from databricks cli, but not from terraform. We need to pass to the terraform binary the NO_PROXY variable as well, and I suspect that's causing the problems - but I have no way of confirming that. The service principal we're using for deployments is assigned admin role on the workspace. How can we further debug this?

Re: Databricks Docker CLI image - how to debug Terraform when deploying Asset Bundle

JacekJacek — Fri, 07 Feb 2025 11:11:55 GMT

OK, as it turns out - in order to bypass proxy we needed to set no_proxy env variable in both upper and lower case (!), like this:

NO_PROXY="adb-xxx.azuredatabricks.net" docker run \ -v %teamcity.build.checkoutDir%:/my-bundle \ -v %teamcity.build.checkoutDir%/.databrickscfg:/root/.databrickscfg \ -w /my-bundle \ -e ARM_CLIENT_SECRET="%env.TF_VAR_client_secret%" \ -e ARM_CLIENT_ID="%env.TF_VAR_client_id%" \ -e ARM_TENANT_ID="%env.TF_VAR_tenant_id%" \ -e NO_PROXY=$NO_PROXY \ -e no_proxy=$NO_PROXY \ %docker.repository%/databricks/cli:0.238.0 bundle deploy -t prototype-dev -p DEFAULT

topic Re: Databricks Docker CLI image - how to debug Terraform when deploying Asset Bundle in Administration & Architecture

Databricks Docker CLI image - how to debug Terraform when deploying Asset Bundle

Re: Databricks Docker CLI image - how to debug Terraform when deploying Asset Bundle