โ07-15-2023 03:49 AM
Hi all!!!
I have created a cluster policy but when i want to use that while creating dlt pipeline, It is showing none. I have checked I have all the necessary permissions to create cluster policies. Still, in dlt ui it is showing none.
โ09-22-2023 11:20 AM
Hi, even i'm facing the same issue. I have a policy with all permissions and it is still not showing up in the DLT pipeline setting UI to select. Please help
โ09-22-2023 12:31 PM - edited โ09-22-2023 12:31 PM
You can check your cluster policy. There is a property called "cluster_type" where you can specify if it will apply to "dlt" or not. https://docs.databricks.com/en/administration-guide/clusters/policies.html
"cluster_type": {
"type": "fixed",
"value": "dlt"
},
โ09-22-2023 12:39 PM
@btafur That helped, the issue is fixed now. Thanks.
a follow up question, can we give 2 values there for cluster type, something like this
"cluster_type": {
"type": "fixed",
"value": "dlt"
},
"cluster_type": {
"type": "fixed",
"value": "job"
}
or comma separated values.
Thanks again!
โ09-22-2023 12:44 PM
You should be able to use "allowlist" instead of "fixed" for multiple values. For example:
"cluster_type": { "type": "allowlist", "defaultValue": "all-purpose", "values": [ "all-purpose", "job", "dlt" ] },
โ09-22-2023 12:46 PM
That's perfect thank you. @btafur
โ09-22-2023 12:53 PM
@btafur Can we also set the auto_terminate minutes with the policy? (for the dlt cluster type)
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group