topic Specifying a serverless cluster for the dev environment in databricks.yml in Data Engineering

Specifying a serverless cluster for the dev environment in databricks.yml

lukasz_wybieral — Tue, 12 Aug 2025 08:45:20 GMT

Hey, I'm trying to find a way to specify a serverless cluster for the dev environment and job clusters for the test and prod environments in databricks.yml.

The problem is that it seems impossible - I’ve tried many approaches, but the only outcomes I can achieve are either:

A. I don’t specify any cluster, and it runs on Serverless for all three environments.
B. I specify a cluster, but then I can’t explicitly set it to be serverless for dev (it must be any of predefined job clusters).

I’ve searched everywhere but haven’t found any information on how to do this.

Has anyone found a way to make this work?

Here’s the code I wish would work (or something similar), so you know what I mean:

#databricks.yml file bundle: name: medallion_architecture include: - resources/*.yml variables: catalog: type: string targets: dev: mode: development default: true workspace: host: adb-xxxxxxxxxxxxxx.azuredatabricks.net variables: catalog: "dev" cluster_config: serverless # (this doesn't work) test: mode: production workspace: host: adb-xxxxxxxxxxxxxx.azuredatabricks.net root_path: /Shared/.bundle/${bundle.target}/${bundle.name} variables: catalog: "test" cluster_config: spark_version: "15.4.x-scala2.12" node_type_id: "Standard_D8s_v3" autoscale: min_workers: 2 max_workers: 6 azure_attributes: availability: "ON_DEMAND_AZURE" prod: mode: production workspace: host: adb-xxxxxxxxxxxxxx.azuredatabricks.net root_path: /Shared/.bundle/${bundle.target}/${bundle.name} run_as: user_name: ${workspace.current_user.userName} variables: catalog: "prod" cluster_config: spark_version: "15.4.x-scala2.12" node_type_id: "Standard_D8s_v3" autoscale: min_workers: 2 max_workers: 10 azure_attributes: availability: "ON_DEMAND_AZURE"

Re: Specifying a serverless cluster for the dev environment in databricks.yml

Nivethan_Venkat — Tue, 12 Aug 2025 09:27:56 GMT

Hi @lukasz_wybieral,

It is not necessary to specify the cluster_config, if you would like to use serverless. Be default, Databricks picks the Serverless cluster if you don't specify the cluster configuration. Attaching below databricks.yml for your reference:

This is my how my job looks like if the cluster configuration is not specified:

But it is good to refer here, for which jobs Serverless computes can be configured: https://docs.databricks.com/aws/en/jobs/compute

Thanks & Regards,
Nivethan V

Re: Specifying a serverless cluster for the dev environment in databricks.yml

lukasz_wybieral — Tue, 12 Aug 2025 10:30:27 GMT

Hey Nivethan, thanks, I’m aware of that, but it doesn’t solve my problem.

I want to be able to define different clusters for each environment, specifically:

Serverless for dev
Medium job cluster for test
Large job cluster for prod

Currently, it’s either:

Define job clusters (expect for Serverless) for all environments, or
Don’t define any at all (and it defaults to Serverless to all targets).

It’s not a huge issue, but it’s annoying.