topic Re: How to set environment (client) on notebook via API/Terraform provider? in Data Engineering

How to set environment (client) on notebook via API/Terraform provider?

korijn — Wed, 08 Jan 2025 16:16:46 GMT

I am deploying a job with a notebook task via the Terraform provider. I want to set the client version to 2. I do NOT need to install any dependencies. I just want to use the new client version for the serverless compute. How do I do this with the Terraform provider or the API? Going through the UI is not acceptable.

It seems impossible:

│ Error: cannot create job: A task environment can not be provided for notebook task config. Please use the %pip magic command to install notebook-scoped Python libraries and Python wheel packages

The error message is not helpful. There are no magic commands that I can run inside a notebook to change the client version.

Re: How to set environment (client) on notebook via API/Terraform provider?

Walter_C — Wed, 08 Jan 2025 16:27:50 GMT

Unfortunately, there is no direct way to set the client version for a notebook task via the Terraform provider or the API without using the UI. The error message suggests that the %pip magic command is the recommended approach for installing dependencies within a notebook, but this does not apply to setting the client version.

Given the constraints, you might need to consider alternative approaches, such as:

Using a different task type: If possible, consider using a different task type that allows setting the client version directly.
Pre-configuring the environment: Set up the environment with the required client version before running the notebook task. This might involve creating a custom cluster with the desired configuration.