Hey, I'm trying to find a way to specify a serverless cluster for the dev environment and job clusters for the test and prod environments in databricks.yml.
The problem is that it seems impossible - Iโve tried many approaches, but the only outcomes I can achieve are either:
A. I donโt specify any cluster, and it runs on Serverless for all three environments.
B. I specify a cluster, but then I canโt explicitly set it to be serverless for dev (it must be any of predefined job clusters).
Iโve searched everywhere but havenโt found any information on how to do this.
Has anyone found a way to make this work?
Hereโs the code I wish would work (or something similar), so you know what I mean:
#databricks.yml file
bundle:
name: medallion_architecture
include:
- resources/*.yml
variables:
catalog:
type: string
targets:
dev:
mode: development
default: true
workspace:
host: adb-xxxxxxxxxxxxxx.azuredatabricks.net
variables:
catalog: "dev"
cluster_config: serverless # (this doesn't work)
test:
mode: production
workspace:
host: adb-xxxxxxxxxxxxxx.azuredatabricks.net
root_path: /Shared/.bundle/${bundle.target}/${bundle.name}
variables:
catalog: "test"
cluster_config:
spark_version: "15.4.x-scala2.12"
node_type_id: "Standard_D8s_v3"
autoscale:
min_workers: 2
max_workers: 6
azure_attributes:
availability: "ON_DEMAND_AZURE"
prod:
mode: production
workspace:
host: adb-xxxxxxxxxxxxxx.azuredatabricks.net
root_path: /Shared/.bundle/${bundle.target}/${bundle.name}
run_as:
user_name: ${workspace.current_user.userName}
variables:
catalog: "prod"
cluster_config:
spark_version: "15.4.x-scala2.12"
node_type_id: "Standard_D8s_v3"
autoscale:
min_workers: 2
max_workers: 10
azure_attributes:
availability: "ON_DEMAND_AZURE"
