cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Databricks Docker CLI image - how to debug Terraform when deploying Asset Bundle

JacekJacek
New Contributor III

We're having issues when deploying asset bundle using docker databricks cli image.

Validation part passes OK:

14:21:15   Name: test
14:21:15   Target: prototype-dev
14:21:15   Workspace:
14:21:15     Host: https://adb-xxx.azuredatabricks.net/
14:21:15     User: ***
14:21:15     Path: /Workspace/Users/***/.bundle/test/prototype-dev
14:21:15   
14:21:15   Validation OK!

And the bundle upload works OK as well:

14:21:20   Uploading bundle files to /Workspace/Users/***/.bundle/test/prototype-dev/files...
14:21:27   Deploying resources...
14:21:29   Updating deployment state...
14:21:30   Deployment complete!

But then when cli runs terraform it fails:

14:21:30   Error: terraform apply: exit status 1
14:21:30   
14:21:30   Error: cannot create job: Unauthorized network access to workspace: xxx
14:21:30   
14:21:30     with databricks_job.configure-file-sources,
14:21:30     on bundle.tf.json line 96, in resource.databricks_job.configure-file-sources:
14:21:30     96:       },
14:21:30   
14:21:30   
14:21:30   Error: cannot create pipeline: Unauthorized network access to workspace: xxx
14:21:30   
14:21:30     with databricks_pipeline.file_ingestion_pipeline,
14:21:30     on bundle.tf.json line 313, in resource.databricks_pipeline.file_ingestion_pipeline:
14:21:30    313:       }
14:21:30   14:21:30   Error: terraform apply: exit status 1
14:21:30   
14:21:30   Error: cannot create job: Unauthorized network access to workspace: xxx
14:21:30   
14:21:30     with databricks_job.configure-file-sources,
14:21:30     on bundle.tf.json line 96, in resource.databricks_job.configure-file-sources:
14:21:30     96:       },
14:21:30   
14:21:30   
14:21:30   Error: cannot create pipeline: Unauthorized network access to workspace: xxx
14:21:30   
14:21:30     with databricks_pipeline.file_ingestion_pipeline,
14:21:30     on bundle.tf.json line 313, in resource.databricks_pipeline.file_ingestion_pipeline:
14:21:30    313:       }
14:21:30   

we are running the deploy command from TeamCity like this:

NO_PROXY="adb-xxx.azuredatabricks.net"

docker run \
  -v %teamcity.build.checkoutDir%:/my-bundle \
  -v %teamcity.build.checkoutDir%/.databrickscfg:/root/.databrickscfg \
  -w /my-bundle \
  -e ARM_CLIENT_SECRET="%env.TF_VAR_client_secret%" \
  -e ARM_CLIENT_ID="%env.TF_VAR_client_id%" \
  -e ARM_TENANT_ID="%env.TF_VAR_tenant_id%" \
  -e NO_PROXY=$NO_PROXY \
  -e TF_LOG="DEBUG" \
  %docker.repository%/databricks/cli:0.238.0 bundle deploy -t prototype-dev -p DEFAULT --debug

And we can see the debugging output from databricks cli, but not from terraform. We need to pass to the terraform binary the NO_PROXY variable as well, and I suspect that's causing the problems - but I have no way of confirming that. The service principal we're using for deployments is assigned admin role on the workspace. How can we further debug this?

1 ACCEPTED SOLUTION

Accepted Solutions

JacekJacek
New Contributor III

OK, as it turns out - in order to bypass proxy we needed to set no_proxy env variable in both upper and lower case (!), like this:

NO_PROXY="adb-xxx.azuredatabricks.net"

docker run \
  -v %teamcity.build.checkoutDir%:/my-bundle \
  -v %teamcity.build.checkoutDir%/.databrickscfg:/root/.databrickscfg \
  -w /my-bundle \
  -e ARM_CLIENT_SECRET="%env.TF_VAR_client_secret%" \
  -e ARM_CLIENT_ID="%env.TF_VAR_client_id%" \
  -e ARM_TENANT_ID="%env.TF_VAR_tenant_id%" \
  -e NO_PROXY=$NO_PROXY \
  -e no_proxy=$NO_PROXY \
  %docker.repository%/databricks/cli:0.238.0 bundle deploy -t prototype-dev -p DEFAULT

View solution in original post

1 REPLY 1

JacekJacek
New Contributor III

OK, as it turns out - in order to bypass proxy we needed to set no_proxy env variable in both upper and lower case (!), like this:

NO_PROXY="adb-xxx.azuredatabricks.net"

docker run \
  -v %teamcity.build.checkoutDir%:/my-bundle \
  -v %teamcity.build.checkoutDir%/.databrickscfg:/root/.databrickscfg \
  -w /my-bundle \
  -e ARM_CLIENT_SECRET="%env.TF_VAR_client_secret%" \
  -e ARM_CLIENT_ID="%env.TF_VAR_client_id%" \
  -e ARM_TENANT_ID="%env.TF_VAR_tenant_id%" \
  -e NO_PROXY=$NO_PROXY \
  -e no_proxy=$NO_PROXY \
  %docker.repository%/databricks/cli:0.238.0 bundle deploy -t prototype-dev -p DEFAULT

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group