Hello,

I have a case where I am executing notebooks from an external system using databricks api /api/2.2/jobs/runs/submit. This has always been non problematic with the job compute, but due to the quite recent serverless for notebooks support being introduced, it was decided to switch to that.

The issue I have is the limitations within serverless environment version 1, as it does not support schema evolution in merge operations, last week serverless environment version 2 was introduced which supports the withSchemaEvolution method of the DeltaMergeBuilder class so it solves the issue i was facing initially, but it introduces a new one as the only way I am able to change the version is through the UI on a notebook level, however once databricks content gets freshly deployed to the workspace, the value is again set to 1, so I don't want to go and everytime after deployment change the value from 1 to 2.

I tried in the api call defining the environment as per documentation and setting the client value to "2":

Also tried to then reference the environment_key defined in environments in the task section, but that returns an error that environments can't be passed to notebook tasks.

My question is if there is any way to programatically specify in the api call that I want to use serverless environment version 2 or somehow else make the version 2 default on workspace level for all notebooks?