topic Re: run a Databricks notebook on serverless environment version 4 with Asset Bundles in Data Engineering

run a Databricks notebook on serverless environment version 4 with Asset Bundles

DatabricksEngi1 — Tue, 16 Sep 2025 06:16:01 GMT

Hi everyone,
I’m working with Databricks Asset Bundles and running jobs that use notebooks (.ipynb).

According to the documentation, it should be possible to set an environment version for serverless jobs. I want to force all of my notebook tasks to run with environment version 4.

• When I omit the cluster definition (to rely on serverless), the job always defaults to environment version 1.
• I don’t select the environment through the UI, since I’m managing everything through DAB YAML.
• Adding an environment_key works for wheel tasks, but not for notebook tasks.

How can I make a .ipynb notebook job run on serverless with environment version 4 using Asset Bundles?
Is there a way to enforce this, or are notebook tasks limited to environment v1 only?

Thanks in advance for any insights.

Re: run a Databricks notebook on serverless environment version 4 with Asset Bundles

szymon_dybczak — Tue, 16 Sep 2025 07:14:47 GMT

Hi @DatabricksEngi1

When you're defining job in DAB you're using job mapping. One of the key of job mapping is called environments. This is the one you're looking for:

Databricks Asset Bundles resources - Azure Databricks | Microsoft Learn

Re: run a Databricks notebook on serverless environment version 4 with Asset Bundles

DatabricksEngi1 — Tue, 16 Sep 2025 08:22:36 GMT

Thank you for your response, but according to the documentation, this does not work with .ipynb files.
My jobs use this extension - would it be possible?

Re: run a Databricks notebook on serverless environment version 4 with Asset Bundles

szymon_dybczak — Tue, 16 Sep 2025 09:01:04 GMT

Hi @DatabricksEngi1 ,

ipynb is a default format for notebooks. Could you provide me a documentation entry where they're saying it's not supported? In my opinion - it should work.

Re: run a Databricks notebook on serverless environment version 4 with Asset Bundles

DatabricksEngi1 — Tue, 16 Sep 2025 09:53:18 GMT

when I try to configure it in the yaml :
resources:
jobs:
ofirtest:
name: ofirtest
tasks:
- task_key: ofirtest
notebook_task:
notebook_path: /Workspace/Users/username/Untitled Notebook
2025-09-15 14:22:51

source: WORKSPACE
environment_key: default
queue:
enabled: true
environments:
- environment_key: default
spec:
environment_version: "4"
performance_target: PERFORMANCE_OPTIMIZED

this is how it looks like:

Re: run a Databricks notebook on serverless environment version 4 with Asset Bundles

szymon_dybczak — Tue, 16 Sep 2025 10:02:35 GMT

Thanks for sharing @DatabricksEngi1. So it seems that currently you can't set this programatically with DAB. I've found official answer of databricks employee to similiar question as your. So unfrotunately, currently serverless environment version must be set at the notebook level through UI

Solved: Programatic selection of serverless compute for no... - Databricks Community - 110225

Re: run a Databricks notebook on serverless environment version 4 with Asset Bundles

DatabricksEngi1 — Tue, 16 Sep 2025 10:05:01 GMT

Thank you very much for your comments.

Hope there will be a solution soon.

Re: run a Databricks notebook on serverless environment version 4 with Asset Bundles

szymon_dybczak — Tue, 16 Sep 2025 10:25:01 GMT

Yes, I think this is something that will be added in the future 🙂