Install notebook dependency via terraform for serverless notebook tasks

rgomez
New Contributor

‎12-23-2024 04:02 PM

I am trying to install a wheel file as a dependency for a serverless notebook task via terraform. According to https://docs.databricks.com/en/compute/serverless/dependencies.html , dependencies in serverless notebooks can be configured via the base environment or the notebook environment. I was taking look at the databricks_job resource in https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job and found the environment configuration block. After taking a closer look at the API documentation  in  https://docs.databricks.com/api/workspace/jobs/create#environments it seems that the environment configuration block is only available for non-notebooks tasks. What would be the best way to configure the base or notebook environment via terraform for serverless notebook tasks? Thanks!

Preview file
185 KB
Preview file
140 KB
0 Kudos

Walter_C
Databricks Employee
Databricks Employee

‎12-23-2024 06:28 PM

Currently, the databricks_job resource in Terraform does not support configuring the environment for notebook tasks directly. You can upload the YAML file and configure the environment as mentioned in https://docs.databricks.com/en/compute/serverless/dependencies.html#configure-a-base-environment via API or CLI and then you will need to set the notebook via the UI

AndriusVitkausk
New Contributor III
In response to Walter_C

‎01-09-2025 05:44 AM

What about applying this through asset bundles? I have the environment definition deployed on a workflow as part of a bundle with multiple notebook tasks...

AndriusVitkausk_0-1736430176728.png

However the dependancy isn't installed on the notebooks, unless i go into the notebook itself, apply the environment, only then do workflows execute as expected, that however raises issues for deployments?