Databricks Community

lukasz_wybieral · ‎08-12-2025

Hey, I'm trying to find a way to specify a serverless cluster for the dev environment and job clusters for the test and prod environments in databricks.yml.

The problem is that it seems impossible - I’ve tried many approaches, but the only outcomes I can achieve are either:

A. I don’t specify any cluster, and it runs on Serverless for all three environments.
B. I specify a cluster, but then I can’t explicitly set it to be serverless for dev (it must be any of predefined job clusters).

I’ve searched everywhere but haven’t found any information on how to do this.

Has anyone found a way to make this work?

Here’s the code I wish would work (or something similar), so you know what I mean:

#databricks.yml file

bundle:
  name: medallion_architecture

include:
  - resources/*.yml

variables:
  catalog:
    type: string

targets:
  dev:
    mode: development
    default: true
    workspace:
      host: adb-xxxxxxxxxxxxxx.azuredatabricks.net
    variables:
      catalog: "dev"
      cluster_config: serverless # (this doesn't work)

  test:
    mode: production
    workspace:
      host: adb-xxxxxxxxxxxxxx.azuredatabricks.net
      root_path: /Shared/.bundle/${bundle.target}/${bundle.name}
    variables:
      catalog: "test"
      cluster_config:
        spark_version: "15.4.x-scala2.12"
        node_type_id: "Standard_D8s_v3"
        autoscale:
          min_workers: 2
          max_workers: 6
        azure_attributes:
          availability: "ON_DEMAND_AZURE"


  prod:
    mode: production
    workspace:
      host: adb-xxxxxxxxxxxxxx.azuredatabricks.net
      root_path: /Shared/.bundle/${bundle.target}/${bundle.name}
    run_as:
      user_name: ${workspace.current_user.userName}
    variables:
      catalog: "prod"
      cluster_config:
        spark_version: "15.4.x-scala2.12"
        node_type_id: "Standard_D8s_v3"
        autoscale:
          min_workers: 2
          max_workers: 10
        azure_attributes:
          availability: "ON_DEMAND_AZURE"

Nivethan_Venkat · ‎08-12-2025

Hi @lukasz_wybieral,

It is not necessary to specify the cluster_config, if you would like to use serverless. Be default, Databricks picks the Serverless cluster if you don't specify the cluster configuration. Attaching below databricks.yml for your reference:

This is my how my job looks like if the cluster configuration is not specified:

But it is good to refer here, for which jobs Serverless computes can be configured: https://docs.databricks.com/aws/en/jobs/compute

Thanks & Regards,
Nivethan V