Compute Policy for DLT pipeline error

iress-nghia — Tue, 16 Jun 2026 01:55:22 GMT

I have a compute policy created using terraform

"unified_archive_pipeline_policy" = { policy_name = "Unified Archive Pipeline Policy" policy_definition = { "cluster_type" = { type = "fixed" value = "dlt" } # Storage-optimized with high-speed local NVMe storage "node_type_id" = { type = "fixed" value = "i4i.xlarge" } # Lower-cost, general-purpose compute driver "driver_node_type_id" = { type = "fixed" value = "m6i.xlarge" } "aws_attributes.first_on_demand" = { type = "fixed" value = 1 } # Fallback protects your ETL from Spot shortages "aws_attributes.availability" = { type = "fixed" value = "SPOT_WITH_FALLBACK" } "aws_attributes.spot_bid_price_percent" = { type = "fixed" value = 100 } "autoscale.min_workers" = { type = "fixed" value = 2 } "autoscale.max_workers" = { type = "fixed" value = 4 } # unified archive bronze workload is incremental append-only; vectorized execution is not needed "runtime_engine" = { type = "fixed" value = "STANDARD" # "STANDARD" turns Photon completely off to save cost hidden = true # Hides the checkbox in the UI to prevent developer confusion } } permissions = { "de-data-insights" = "CAN_USE" } }

When I deploy my pipeline that refer to this policy like this

clusters: - label: default policy_id: ${var.unified_archive_pipeline_cluster_policy} # References the dynamic lookup variable apply_policy_default_values: true

I got error related to data type

INVALID_PARAMETER_VALUE: [DLT ERROR CODE: INVALID_CLUSTER_SETTING.CLIENT_ERROR] The resolved settings for the 'updates' cluster are not compatible with the configured cluster policy because of the following failure: INVALID_PARAMETER_VALUE: Validation failed for autoscale.min_workers, the value must be 2 (is "2"); Validation failed for autoscale.max_workers, the value must be 4 (is "4"); Validation failed for aws_attributes.first_on_demand, the value must be 1 (is "1"); Validation failed for aws_attributes.spot_bid_price_percent, the value must be 100 (is "100") To fix this error: 1. Check whether the incompatible settings are part of the 'updates' or 'default' cluster label in the pipeline settings. If so, please update the pipeline settings with compatible values. 2. Delta Live Tables applies default settings for some cluster attributes if they are unspecified by the user. In this case, the default cluster settings provided by Delta Live Tables might not be compatible with the configured cluster policy. To fix this error, please set corresponding cluster attributes with compatible settings for the 'updates' cluster in your pipeline settings to prevent DLT default cluster settings from getting applied. 3. If these efforts do not resolve the error, contact Databricks support.

topic Compute Policy for DLT pipeline error in Data Engineering

Compute Policy for DLT pipeline error