topic Re: Programatic selection of serverless compute for notebooks environment version in Get Started Discussions

Programatic selection of serverless compute for notebooks environment version

tts — Fri, 14 Feb 2025 13:35:20 GMT

Hello,

I have a case where I am executing notebooks from an external system using databricks api /api/2.2/jobs/runs/submit. This has always been non problematic with the job compute, but due to the quite recent serverless for notebooks support being introduced, it was decided to switch to that.

The issue I have is the limitations within serverless environment version 1, as it does not support schema evolution in merge operations, last week serverless environment version 2 was introduced which supports the withSchemaEvolution method of the DeltaMergeBuilder class so it solves the issue i was facing initially, but it introduces a new one as the only way I am able to change the version is through the UI on a notebook level, however once databricks content gets freshly deployed to the workspace, the value is again set to 1, so I don't want to go and everytime after deployment change the value from 1 to 2.

I tried in the api call defining the environment as per documentation and setting the client value to "2":

Also tried to then reference the environment_key defined in environments in the task section, but that returns an error that environments can't be passed to notebook tasks.

My question is if there is any way to programatically specify in the api call that I want to use serverless environment version 2 or somehow else make the version 2 default on workspace level for all notebooks?

Re: Programatic selection of serverless compute for notebooks environment version

Alberto_Umana — Sun, 16 Feb 2025 00:40:28 GMT

Hello @tts,

there is no way to programmatically specify the version of the serverless environment for notebook tasks in the Databricks API call /api/2.2/jobs/runs/submit. Currently, serverless environment versions must be set at the notebook level through the UI, and this setting gets reset to version 1 every time Databricks content is deployed to the workspace. I will check internally if there is a workaround for now.

Re: Programatic selection of serverless compute for notebooks environment version

tts — Mon, 17 Feb 2025 06:25:13 GMT

Hello @Alberto_Umana,

Thank you for the response. Let me know if you find any way to handle this or if this is atleast something that will be available in the future.

Re: Programatic selection of serverless compute for notebooks environment version

tts — Thu, 20 Feb 2025 15:44:20 GMT

Hi @Alberto_Umana,

Has there been any updates regarding this internally?

Re: Programatic selection of serverless compute for notebooks environment version

Alberto_Umana — Thu, 20 Feb 2025 15:55:24 GMT

Hi @tts,

Thanks for following up.. I noticed that serverless version 2 is now default version, are you still hitting the failure?

Re: Programatic selection of serverless compute for notebooks environment version

tts — Thu, 20 Feb 2025 15:59:16 GMT

Did not notice that, but you are right, thanks for the follow and for your help.

Re: Programatic selection of serverless compute for notebooks environment version

JakubSkibicki — Fri, 21 Feb 2025 15:46:18 GMT

As an alternative environment for Serverless could be set in asset bundle job configuration.

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/jobs-tutorial#configure-a-job-that-uses-serverless-compute

Re: Programatic selection of serverless compute for notebooks environment version

tts — Fri, 28 Feb 2025 12:02:42 GMT

This only applies to Python scripts, Python wheesl and dbt tasks, but in my case I want to have possibility to set it for notebook tasks as well. Currently, when notebooks are deployed with asset bundles, the value still gets set to 1. If I create a new notebook in the workspace, the environment version is then 2. So what I struggle to find is how to set the value for notebook tasks or if that is even possible at the moment.

Re: Programatic selection of serverless compute for notebooks environment version

GerardW — Thu, 29 May 2025 02:35:48 GMT

@tts, did you manage to get this resolved? Battling the same issue here and cannot find a way to make it work. Even though the default API is 2 now, it still seems to randomly assign an API when deploy via DAB. @JakubSkibicki, tried this too but hitting the same issue as @tts., cannot be used with notebook tasks curently.

Re: Programatic selection of serverless compute for notebooks environment version

toby_chu — Sun, 30 Nov 2025 23:00:43 GMT

Not so sure about the general but in eu-west-3, we could specify the serverless environment version using DAB using the `environments` block and `spec` params:

resources: jobs: pipeline: name: "[${bundle.target}]pipeline" webhook_notifications: ${var.webhook_notifications_code_block} health: rules: - {metric: RUN_DURATION_SECONDS, op: GREATER_THAN, value: 1200} timeout_seconds: 3600 queue: enabled: true tasks: - task_key: task-key notebook_task: notebook_path: "${var.notebook_path}" environment_key: default disable_auto_optimization: true max_retries: 0 environments: - environment_key: default spec: client: "4" parameters: - name: env_file_path default: "${var.env_file_path}"