- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-20-2026 04:46 AM
Hello,
I am building a pipeline using DAB.
The first step of the dab is to deploy my library as a wheel.
The pipeline is run on a shared databricks cluster.
When I run the job I see that the job is not using exactly the requirements I specified but it used the versions that were already in the cluster.
Is there a way to have a specific environment for my job using this shared cluster ?
- Labels:
-
Model Serving
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-22-2026 02:35 PM
This is the nature of shared clusters . You can install libraries for a task but isolation is not guaranteed . if a library is already installed on the cluster it will take priority over what defined for the task.
Any reason you cant use job clusters . They are cheap and provide the isolation needed in your case . Another options is using Serverless jobs with environment specs .
Pradeep Singh - https://www.linkedin.com/in/dbxdev
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-22-2026 06:40 PM
Hi @Dali1,
+1 to @pradeep_singh, on shared clusters, tasks inherit cluster-installed libraries, so you won’t get a clean, versioned environment. Use a job cluster (new_cluster) or switch to serverless jobs with an environment per task for isolation. With serverless, define job.environments and set environment_key on your wheel/script task. List your wheel/requirements in dependencies to pin exact version.
