cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to set environment (client) on notebook via API/Terraform provider?

korijn
New Contributor II

I am deploying a job with a notebook task via the Terraform provider. I want to set the client version to 2. I do NOT need to install any dependencies. I just want to use the new client version for the serverless compute. How do I do this with the Terraform provider or the API? Going through the UI is not acceptable.

It seems impossible:

│ Error: cannot create job: A task environment can not be provided for notebook task config. Please use the %pip magic command to install notebook-scoped Python libraries and Python wheel packages

The error message is not helpful. There are no magic commands that I can run inside a notebook to change the client version.

1 REPLY 1

Walter_C
Databricks Employee
Databricks Employee

Unfortunately, there is no direct way to set the client version for a notebook task via the Terraform provider or the API without using the UI. The error message suggests that the %pip magic command is the recommended approach for installing dependencies within a notebook, but this does not apply to setting the client version.

Given the constraints, you might need to consider alternative approaches, such as:

  1. Using a different task type: If possible, consider using a different task type that allows setting the client version directly.
  2. Pre-configuring the environment: Set up the environment with the required client version before running the notebook task. This might involve creating a custom cluster with the desired configuration.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group