topic Re: Install notebook dependency via terraform for serverless notebook tasks in Data Engineering

Install notebook dependency via terraform for serverless notebook tasks

rgomez — Tue, 24 Dec 2024 00:02:49 GMT

I am trying to install a wheel file as a dependency for a serverless notebook task via terraform. According to https://docs.databricks.com/en/compute/serverless/dependencies.html , dependencies in serverless notebooks can be configured via the base environment or the notebook environment. I was taking look at the databricks_job resource in https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/job and found the environment configuration block. After taking a closer look at the API documentation in https://docs.databricks.com/api/workspace/jobs/create#environments it seems that the environment configuration block is only available for non-notebooks tasks. What would be the best way to configure the base or notebook environment via terraform for serverless notebook tasks? Thanks!

Re: Install notebook dependency via terraform for serverless notebook tasks

Walter_C — Tue, 24 Dec 2024 02:28:29 GMT

Currently, the databricks_job resource in Terraform does not support configuring the environment for notebook tasks directly. You can upload the YAML file and configure the environment as mentioned in https://docs.databricks.com/en/compute/serverless/dependencies.html#configure-a-base-environment via API or CLI and then you will need to set the notebook via the UI

Re: Install notebook dependency via terraform for serverless notebook tasks

AndriusVitkausk — Thu, 09 Jan 2025 13:44:05 GMT

What about applying this through asset bundles? I have the environment definition deployed on a workflow as part of a bundle with multiple notebook tasks...

However the dependancy isn't installed on the notebooks, unless i go into the notebook itself, apply the environment, only then do workflows execute as expected, that however raises issues for deployments?