Policy for DLT
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-04-2025 01:52 PM
Hi,
I am trying to define a policy for our DLT pipelines and I would like to provide a specific spark version like in the below example:
{
"spark_conf.spark.databricks.cluster.profile": {
"type": "forbidden",
"hidden": true
},
"spark_version": {
"type": "allowlist",
"values": [
"14.3.x-scala2.12"
]
},
"node_type_id": {
"type": "unlimited",
"defaultValue": "Standard_DS3_v2",
"isOptional": true
},
"num_workers": {
"type": "unlimited",
"defaultValue": 4,
"isOptional": true
},
"azure_attributes.availability": {
"type": "unlimited",
"defaultValue": "SPOT_WITH_FALLBACK_AZURE"
},
"azure_attributes.spot_bid_max_price": {
"type": "fixed",
"value": 100,
"hidden": true
},
"instance_pool_id": {
"type": "forbidden",
"hidden": true
},
"driver_instance_pool_id": {
"type": "forbidden",
"hidden": true
},
"cluster_type": {
"type": "fixed",
"value": "dlt"
}
}
But I am getting this error in my pipeline:
INVALID_PARAMETER_VALUE: [DLT ERROR CODE: INVALID_CLUSTER_SETTING.CLIENT_ERROR] The cluster policy specified in the pipeline settings is not compatible with Delta Live Tables. Remove 'spark_version’ from your cluster policy.
Could you please help me with it?