topic Re: Cluster policy not showing while creating delta live table pipeline in Get Started Discussions

Cluster policy not showing while creating delta live table pipeline

Mbinyala — Sat, 15 Jul 2023 10:49:19 GMT

Hi all!!!

I have created a cluster policy but when i want to use that while creating dlt pipeline, It is showing none. I have checked I have all the necessary permissions to create cluster policies. Still, in dlt ui it is showing none.

Re: Cluster policy not showing while creating delta live table pipeline

Rishitha — Fri, 22 Sep 2023 18:20:08 GMT

Hi, even i'm facing the same issue. I have a policy with all permissions and it is still not showing up in the DLT pipeline setting UI to select. Please help

Re: Cluster policy not showing while creating delta live table pipeline

btafur — Fri, 22 Sep 2023 19:31:25 GMT

You can check your cluster policy. There is a property called "cluster_type" where you can specify if it will apply to "dlt" or not. https://docs.databricks.com/en/administration-guide/clusters/policies.html

  "cluster_type": {
    "type": "fixed",
    "value": "dlt"
  },

Re: Cluster policy not showing while creating delta live table pipeline

Rishitha — Fri, 22 Sep 2023 19:39:19 GMT

@btafur That helped, the issue is fixed now. Thanks.

a follow up question, can we give 2 values there for cluster type, something like this

"cluster_type": {
    "type": "fixed",
    "value": "dlt"
  },
"cluster_type": { 
    "type": "fixed", 
    "value": "job" 
}

or comma separated values.

Thanks again!

Re: Cluster policy not showing while creating delta live table pipeline

btafur — Fri, 22 Sep 2023 19:44:28 GMT

You should be able to use "allowlist" instead of "fixed" for multiple values. For example:

 "cluster_type": {
  "type": "allowlist",
  "defaultValue": "all-purpose",
  "values": [
   "all-purpose",
   "job",
   "dlt"
  ]
 },

Re: Cluster policy not showing while creating delta live table pipeline

Rishitha — Fri, 22 Sep 2023 19:46:02 GMT

That's perfect thank you. @btafur

Re: Cluster policy not showing while creating delta live table pipeline

Rishitha — Fri, 22 Sep 2023 19:53:03 GMT

@btafur Can we also set the auto_terminate minutes with the policy? (for the dlt cluster type)