Libraries for Serverless Compute

[akshay_prabhu31]

Current Setup:
I'm working on a project using the Databricks Free Trial, and I'm running my workloads on serverless compute.

Issue:
Every time I run a pipeline or job, it takes 30 minutes or more to complete. This delay is mainly due to the repeated installation of libraries in each notebook that's part of the job.

Question:
Is there a way to pre-install or attach the required libraries to the serverless compute environment—similar to setting up an environment—so that the libraries don't need to be reinstalled during every job run?

[Advika]

Hello @akshay_prabhu31!

To manage dependencies on Databricks serverless compute, you can use the Environment side panel in the notebook to add the required Python libraries.

You can refer to this documentation: https://docs.databricks.com/gcp/en/compute/serverless/dependencies