Databricks Docker CLI image - how to debug Terraform when deploying Asset Bundle

[JacekJacek]

We're having issues when deploying asset bundle using docker databricks cli image.

Validation part passes OK:

14:21:15   Name: test
14:21:15   Target: prototype-dev
14:21:15   Workspace:
14:21:15     Host: https://adb-xxx.azuredatabricks.net/
14:21:15     User: ***
14:21:15     Path: /Workspace/Users/***/.bundle/test/prototype-dev
14:21:15   
14:21:15   Validation OK!

And the bundle upload works OK as well:

14:21:20   Uploading bundle files to /Workspace/Users/***/.bundle/test/prototype-dev/files...
14:21:27   Deploying resources...
14:21:29   Updating deployment state...
14:21:30   Deployment complete!

But then when cli runs terraform it fails:

14:21:30   Error: terraform apply: exit status 1
14:21:30   
14:21:30   Error: cannot create job: Unauthorized network access to workspace: xxx
14:21:30   
14:21:30     with databricks_job.configure-file-sources,
14:21:30     on bundle.tf.json line 96, in resource.databricks_job.configure-file-sources:
14:21:30     96:       },
14:21:30   
14:21:30   
14:21:30   Error: cannot create pipeline: Unauthorized network access to workspace: xxx
14:21:30   
14:21:30     with databricks_pipeline.file_ingestion_pipeline,
14:21:30     on bundle.tf.json line 313, in resource.databricks_pipeline.file_ingestion_pipeline:
14:21:30    313:       }
14:21:30   14:21:30   Error: terraform apply: exit status 1
14:21:30   
14:21:30   Error: cannot create job: Unauthorized network access to workspace: xxx
14:21:30   
14:21:30     with databricks_job.configure-file-sources,
14:21:30     on bundle.tf.json line 96, in resource.databricks_job.configure-file-sources:
14:21:30     96:       },
14:21:30   
14:21:30   
14:21:30   Error: cannot create pipeline: Unauthorized network access to workspace: xxx
14:21:30   
14:21:30     with databricks_pipeline.file_ingestion_pipeline,
14:21:30     on bundle.tf.json line 313, in resource.databricks_pipeline.file_ingestion_pipeline:
14:21:30    313:       }
14:21:30   

we are running the deploy command from TeamCity like this:

NO_PROXY="adb-xxx.azuredatabricks.net"

docker run \
  -v %teamcity.build.checkoutDir%:/my-bundle \
  -v %teamcity.build.checkoutDir%/.databrickscfg:/root/.databrickscfg \
  -w /my-bundle \
  -e ARM_CLIENT_SECRET="%env.TF_VAR_client_secret%" \
  -e ARM_CLIENT_ID="%env.TF_VAR_client_id%" \
  -e ARM_TENANT_ID="%env.TF_VAR_tenant_id%" \
  -e NO_PROXY=$NO_PROXY \
  -e TF_LOG="DEBUG" \
  %docker.repository%/databricks/cli:0.238.0 bundle deploy -t prototype-dev -p DEFAULT --debug

And we can see the debugging output from databricks cli, but not from terraform. We need to pass to the terraform binary the NO_PROXY variable as well, and I suspect that's causing the problems - but I have no way of confirming that. The service principal we're using for deployments is assigned admin role on the workspace. How can we further debug this?

[JacekJacek · Accepted solution]

OK, as it turns out - in order to bypass proxy we needed to set no_proxy env variable in both upper and lower case (!), like this:

NO_PROXY="adb-xxx.azuredatabricks.net"

docker run \
  -v %teamcity.build.checkoutDir%:/my-bundle \
  -v %teamcity.build.checkoutDir%/.databrickscfg:/root/.databrickscfg \
  -w /my-bundle \
  -e ARM_CLIENT_SECRET="%env.TF_VAR_client_secret%" \
  -e ARM_CLIENT_ID="%env.TF_VAR_client_id%" \
  -e ARM_TENANT_ID="%env.TF_VAR_tenant_id%" \
  -e NO_PROXY=$NO_PROXY \
  -e no_proxy=$NO_PROXY \
  %docker.repository%/databricks/cli:0.238.0 bundle deploy -t prototype-dev -p DEFAULT