Serverless environment not respecting environment spec on run_job_task

[elliottatreef]

When running a job via a `run_job_task`, the job triggered is not using the specified serverless environment. I've configured my job to use serverless `environment_version` "3" with a dependency built into my workspace, but whenever I run the job, it uses env version "2".

In addition, I've tried configuring the default environment for serverless in my workspace to use the same setting as my job's configured runtime and that doesn't work either.

Here, you can see that the job is using version "2".

Even though it is configured with version "3".

[MuthuLakshmi]

@elliottatreef 
Can you try to set the Environment version on the source notebook and then trigger the job?

On notebook -> Serverless -> configuration -> Environment version drop down. 
Then, in your job, making sure it’s assigning to the Serverless compute that you configured. I checked with a simple repro in my local and once we set the Environment version in Notebook it picks the same in Job. 

[Khaja_Zaffer]

Hello @elliottatreef 

Its a known limitation. serverless environments, including the environment_version specification, cannot be applied programmatically via Databricks Asset Bundles (DAB) YAML to notebook_task types.

UI Configuration for Environment Version (Quick but Manual):

Open your notebook in the Databricks workspace. In the top-right compute selector, choose Serverless and set Environment version: 3. Save the notebook.

Job runs (including those triggered via run_job_task) will now use version 3.

For your wheel dependency, add %pip install /Workspace/AssetBundles/data-eng-data-ops/files/dist/data_eng_data_ops-0.1.0-py3-none-any.whl at the top of the notebook.

Drawback: Not automated for CI/CD; requires UI intervention post-DAB deployment.

[elliottatreef]

Thanks for the tip! Funny enough, the pip install is a thing I was already doing, but was trying to clean those up because it looks ugly and isn't something I want at the top of all of my jobs. Does this basically mean that setting an `environment` on a job that is only a notebook task does nothing and I shouldn't even set it?