Install notebook dependency via terraform for serverless notebook tasks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-23-2024 04:02 PM
I am trying to install a wheel file as a dependency for a serverless notebook task via terraform. According to https://docs.databricks.com/en/compute/serverless/dependencies.html , dependencies in serverless notebooks can be configured via the base environment or the notebook environment. I was taking look at the databricks_job resource in https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job and found the environment configuration block. After taking a closer look at the API documentation in https://docs.databricks.com/api/workspace/jobs/create#environments it seems that the environment configuration block is only available for non-notebooks tasks. What would be the best way to configure the base or notebook environment via terraform for serverless notebook tasks? Thanks!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-23-2024 06:28 PM
Currently, the databricks_job
resource in Terraform does not support configuring the environment for notebook tasks directly. You can upload the YAML file and configure the environment as mentioned in https://docs.databricks.com/en/compute/serverless/dependencies.html#configure-a-base-environment via API or CLI and then you will need to set the notebook via the UI
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-09-2025 05:44 AM
What about applying this through asset bundles? I have the environment definition deployed on a workflow as part of a bundle with multiple notebook tasks...
However the dependancy isn't installed on the notebooks, unless i go into the notebook itself, apply the environment, only then do workflows execute as expected, that however raises issues for deployments?

