Serverless environment not respecting environment spec on run_job_task

elliottatreef
New Contributor II

‎10-15-2025 11:53 AM

When running a job via a `run_job_task`, the job triggered is not using the specified serverless environment. I've configured my job to use serverless `environment_version` "3" with a dependency built into my workspace, but whenever I run the job, it uses env version "2".

In addition, I've tried configuring the default environment for serverless in my workspace to use the same setting as my job's configured runtime and that doesn't work either.

Here, you can see that the job is using version "2".

Screenshot 2025-10-15 at 11.40.45 AM.png

Even though it is configured with version "3".

Screenshot 2025-10-15 at 11.43.39 AM.png

0 Kudos

MuthuLakshmi
Databricks Employee
Databricks Employee

‎10-16-2025 12:59 AM

@elliottatreef 
Can you try to set the Environment version on the source notebook and then trigger the job?

On notebook -> Serverless -> configuration -> Environment version drop down. 
Then, in your job, making sure it’s assigning to the Serverless compute that you configured. I checked with a simple repro in my local and once we set the Environment version in Notebook it picks the same in Job. 

0 Kudos

Khaja_Zaffer
Esteemed Contributor
In response to MuthuLakshmi

‎10-16-2025 01:06 AM

Hello @elliottatreef 

Its a known limitation. serverless environments, including the environment_version specification, cannot be applied programmatically via Databricks Asset Bundles (DAB) YAML to notebook_task types.

 

UI Configuration for Environment Version (Quick but Manual):

Open your notebook in the Databricks workspace. In the top-right compute selector, choose Serverless and set Environment version: 3. Save the notebook.

Job runs (including those triggered via run_job_task) will now use version 3.

For your wheel dependency, add %pip install /Workspace/AssetBundles/data-eng-data-ops/files/dist/data_eng_data_ops-0.1.0-py3-none-any.whl at the top of the notebook.

Drawback: Not automated for CI/CD; requires UI intervention post-DAB deployment.

 

Preview file
44 KB

elliottatreef
New Contributor II
In response to Khaja_Zaffer

‎10-16-2025 11:36 AM

Thanks for the tip! Funny enough, the pip install is a thing I was already doing, but was trying to clean those up because it looks ugly and isn't something I want at the top of all of my jobs. Does this basically mean that setting an `environment` on a job that is only a notebook task does nothing and I shouldn't even set it?

0 Kudos