[Databricks Asset Bundles] Bug: driver_node_type_id not updated

[Marthinus]

Working with databricks asset bundles (using the new python-based definition), if you have a job_cluster defined using driver_node_type_id, and then update it to no longer have it defined, but only node_type_id, the driver node_type never gets updated.

[dkushari]

Hi @Marthinus - This is the expected behavior. please refer to the doc here. Please let me know if you have any additional questions.

[Marthinus]

@dkushari no I don't think you understand me. When I unset driver_node_type_id, it does NOT revert to node_type_id.

Perhaps an example will make it clear:
First define cluster as:

JobCluster(
    job_cluster_key="example",
    new_cluster=ClusterSpec(
        num_workers=1,
        node_type_id="Standard_D4ads_v5",
        driver_node_type_id="Standard_E8ads_v5",
    ),
)

Which results in the expected cluster, then edit to following and deploy again:

JobCluster(
    job_cluster_key="example",
    new_cluster=ClusterSpec(
        num_workers=1,
        node_type_id="Standard_D4ads_v5",
    ),
)

And the driver_node_type_id will stay a Standard_E8ads_v5 instead of reverting to a Standard_D4ads_v5 as expected.

[dkushari]

Thanks for the details. The way it works is once you set it when you first define the job cluster it does not change when you later remove driver_node_type_id from the spec (i.e., omit it), Databricks does not automatically revert it to match the node_type_id.

That’s because the Databricks Jobs API treats unset fields as “leave existing value unchanged”, not “reset to default.” This is by design — the API performs a patch-style update, not a full replace.

[Chiran-Gajula]

There is no built-in way in Databricks Asset bundles or terraform to automatically inherit the value of driver_node_type_id for node_type_id.
"You must set both explicitly in your configuration"

You can always see your updated detail resource from the terraform state what its applied. (As Databricks Asset bundle managed by terraform in backend).
if you are in Linux follow below steps to dig more based on your situation.
cd ~/.databricks/bundle/<env>/terraform
Under this you will see bundle.tf.json anf teraform.tfstate file.
You can dig to understand the resource with 
#terraform state show databricks_job.<resource>