Hey folks,

Facing an issue with zone_id not getting overridden when redeploying the DAB template to Databricks workspace.

The Databricks job is already deployed and has "ap-south-1a" zone_id. I wanted to make it "auto" so, I have made the changes to the aws_attributes in the .yml and deployed it via GitHub CI/CD the changes aren't getting reflected in compute but I can see the files have been updated in the workspace.

If I want to change from "ap-south-1a" to "ap-south-1b" it is working but it is not updating for "auto". I am not sure why.

Below is the config

      job_clusters:
        - job_cluster_key: job_cluster
          new_cluster:
            spark_version: 12.2.x-cpu-ml-scala2.12
            spark_conf:
              spark.master: local[*, 4]
            aws_attributes:
              availability: SPOT_WITH_FALLBACK
              zone_id: auto
              spot_bid_price_percent: 90
              ebs_volume_type: GENERAL_PURPOSE_SSD
              ebs_volume_count: 1
              ebs_volume_size: 40
            node_type_id: i3.xlarge
            enable_elastic_disk: false
            policy_id: BC60EEE08F0002CE
            data_security_mode: SINGLE_USER
            runtime_engine: STANDARD
            num_workers: 0

Attached is the screenshot where I am expecting the DAB template/bundle changes to reflect.

I found a workaround as below but it is not scalable hence looking for a solution
- Disconnect from source
- Update zone_id manually
- Redeploy the bundle to connect the job and Git