Compute Policy for DLT pipeline error
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2 weeks ago
I have a compute policy created using terraform
"unified_archive_pipeline_policy" = {
policy_name = "Unified Archive Pipeline Policy"
policy_definition = {
"cluster_type" = {
type = "fixed"
value = "dlt"
}
# Storage-optimized with high-speed local NVMe storage
"node_type_id" = {
type = "fixed"
value = "i4i.xlarge"
}
# Lower-cost, general-purpose compute driver
"driver_node_type_id" = {
type = "fixed"
value = "m6i.xlarge"
}
"aws_attributes.first_on_demand" = {
type = "fixed"
value = 1
}
# Fallback protects your ETL from Spot shortages
"aws_attributes.availability" = {
type = "fixed"
value = "SPOT_WITH_FALLBACK"
}
"aws_attributes.spot_bid_price_percent" = {
type = "fixed"
value = 100
}
"autoscale.min_workers" = {
type = "fixed"
value = 2
}
"autoscale.max_workers" = {
type = "fixed"
value = 4
}
# unified archive bronze workload is incremental append-only; vectorized execution is not needed
"runtime_engine" = {
type = "fixed"
value = "STANDARD" # "STANDARD" turns Photon completely off to save cost
hidden = true # Hides the checkbox in the UI to prevent developer confusion
}
}
permissions = {
"de-data-insights" = "CAN_USE"
}
}When I deploy my pipeline that refer to this policy like this
clusters:
- label: default
policy_id: ${var.unified_archive_pipeline_cluster_policy} # References the dynamic lookup variable
apply_policy_default_values: true
I got error related to data type
INVALID_PARAMETER_VALUE: [DLT ERROR CODE: INVALID_CLUSTER_SETTING.CLIENT_ERROR] The resolved settings for the 'updates' cluster are not compatible with the configured cluster policy because of the following failure:
INVALID_PARAMETER_VALUE: Validation failed for autoscale.min_workers, the value must be 2 (is "2"); Validation failed for autoscale.max_workers, the value must be 4 (is "4"); Validation failed for aws_attributes.first_on_demand, the value must be 1 (is "1"); Validation failed for aws_attributes.spot_bid_price_percent, the value must be 100 (is "100")
To fix this error:
1. Check whether the incompatible settings are part of the 'updates' or 'default' cluster label in the pipeline settings. If so, please update the pipeline settings with compatible values.
2. Delta Live Tables applies default settings for some cluster attributes if they are unspecified by the user. In this case, the default cluster settings provided by Delta Live Tables might not be compatible with the configured cluster policy. To fix this error, please set corresponding cluster attributes with compatible settings for the 'updates' cluster in your pipeline settings to prevent DLT default cluster settings from getting applied.
3. If these efforts do not resolve the error, contact Databricks support.