Compute Policy for DLT pipeline error

[iress-nghia]

I have a compute policy created using terraform

"unified_archive_pipeline_policy" = {
    policy_name = "Unified Archive Pipeline Policy"
    policy_definition = {
      "cluster_type" = {
        type  = "fixed"
        value = "dlt"
      }
      # Storage-optimized with high-speed local NVMe storage
      "node_type_id" = {
        type  = "fixed"
        value = "i4i.xlarge"
      }
      # Lower-cost, general-purpose compute driver
      "driver_node_type_id" = {
        type  = "fixed"
        value = "m6i.xlarge"
      }
      "aws_attributes.first_on_demand" = {
        type  = "fixed"
        value = 1
      }
      # Fallback protects your ETL from Spot shortages
      "aws_attributes.availability" = {
        type  = "fixed"
        value = "SPOT_WITH_FALLBACK"
      }
      "aws_attributes.spot_bid_price_percent" = {
        type  = "fixed"
        value = 100
      }
      "autoscale.min_workers" = {
        type  = "fixed"
        value = 2
      }
      "autoscale.max_workers" = {
        type  = "fixed"
        value = 4
      }
      # unified archive bronze workload is incremental append-only; vectorized execution is not needed
      "runtime_engine" = {
        type   = "fixed"
        value  = "STANDARD" # "STANDARD" turns Photon completely off to save cost
        hidden = true       # Hides the checkbox in the UI to prevent developer confusion
      }
    }
    permissions = {
      "de-data-insights" = "CAN_USE"
    }
  }

When I deploy my pipeline that refer to this policy like this

      clusters:
      - label: default
        policy_id: ${var.unified_archive_pipeline_cluster_policy} # References the dynamic lookup variable
        apply_policy_default_values: true
        

I got error related to data type

INVALID_PARAMETER_VALUE: [DLT ERROR CODE: INVALID_CLUSTER_SETTING.CLIENT_ERROR] The resolved settings for the 'updates' cluster are not compatible with the configured cluster policy because of the following failure:

INVALID_PARAMETER_VALUE: Validation failed for autoscale.min_workers, the value must be 2 (is "2"); Validation failed for autoscale.max_workers, the value must be 4 (is "4"); Validation failed for aws_attributes.first_on_demand, the value must be 1 (is "1"); Validation failed for aws_attributes.spot_bid_price_percent, the value must be 100 (is "100")

To fix this error:
1. Check whether the incompatible settings are part of the 'updates' or 'default' cluster label in the pipeline settings. If so, please update the pipeline settings with compatible values.
2. Delta Live Tables applies default settings for some cluster attributes if they are unspecified by the user. In this case, the default cluster settings provided by Delta Live Tables might not be compatible with the configured cluster policy. To fix this error, please set corresponding cluster attributes with compatible settings for the 'updates' cluster in your pipeline settings to prevent DLT default cluster settings from getting applied.
3. If these efforts do not resolve the error, contact Databricks support.

[binlogreader]

@iress-nghia From the logs, looks like every failure is a numeric field: autoscale.min_workers, autoscale.max_workers, aws_attributes.first_on_demand, aws_attributes.spot_bid_price_percent.

On the contrary, every string field passed: node_type_id, availability, runtime_engine, cluster_type. And the message itself says it: the value must be 2 (is "2"). The number your policy is supposed to hold arrived as the string "2". This is not a DLT problem and your pipeline YAML is fine. It seems to be the the numbers got converted into string on the way through Terraform.

You can confirm this by trying this command:

```
databricks cluster-policies get <policy-id>
```

Look at the definition it returns. If you see `"value": "2"` with quotes on the numeric fields, that's the likely root cause.

We run our policies through Terraform too, and the reason we've never hit this is the shape of the input. We keep each policy definition as a raw .json file with real numbers and booleans in it, load it with `jsondecode(file(...))`, and pass it into a module variable that has no type constraint. jsondecode preserves number and bool types, and an unconstrained (or `type = any`) variable passes them through without unification, so `jsonencode` writes them back out as numbers.

So you have two clean fixes, either works:

1. Change the module variable that receives `policy_definition` to `type = any` (or drop the constraint). Terraform then keeps each entry's exact types and your existing HCL works as written.
2. Move the definition into a plain JSON file and load it with `jsondecode(file("policies/unified_archive.json"))`. This also makes the policy diffable against what the API returns, which is handy.