run a Databricks notebook on serverless environment version 4 with Asset Bundles

DatabricksEngi1
Contributor

Hi everyone,
I’m working with Databricks Asset Bundles and running jobs that use notebooks (.ipynb).

According to the documentation, it should be possible to set an environment version for serverless jobs. I want to force all of my notebook tasks to run with environment version 4.

• When I omit the cluster definition (to rely on serverless), the job always defaults to environment version 1.
• I don’t select the environment through the UI, since I’m managing everything through DAB YAML.
• Adding an environment_key works for wheel tasks, but not for notebook tasks.

How can I make a .ipynb notebook job run on serverless with environment version 4 using Asset Bundles?
Is there a way to enforce this, or are notebook tasks limited to environment v1 only?

Thanks in advance for any insights.