Hey folks,
Facing an issue with zone_id not getting overridden when redeploying the DAB template to Databricks workspace.
The Databricks job is already deployed and has "ap-south-1a" zone_id. I wanted to make it "auto" so, I have made the changes to the aws_attributes in the .yml and deployed it via GitHub CI/CD the changes aren't getting reflected in compute but I can see the files have been updated in the workspace.
If I want to change from "ap-south-1a" to "ap-south-1b" it is working but it is not updating for "auto". I am not sure why.
Below is the config
job_clusters:
- job_cluster_key: job_cluster
new_cluster:
spark_version: 12.2.x-cpu-ml-scala2.12
spark_conf:
spark.master: local[*, 4]
aws_attributes:
availability: SPOT_WITH_FALLBACK
zone_id: auto
spot_bid_price_percent: 90
ebs_volume_type: GENERAL_PURPOSE_SSD
ebs_volume_count: 1
ebs_volume_size: 40
node_type_id: i3.xlarge
enable_elastic_disk: false
policy_id: BC60EEE08F0002CE
data_security_mode: SINGLE_USER
runtime_engine: STANDARD
num_workers: 0
Attached is the screenshot where I am expecting the DAB template/bundle changes to reflect.
I found a workaround as below but it is not scalable hence looking for a solution
- Disconnect from source
- Update zone_id manually
- Redeploy the bundle to connect the job and Git
