Unable to configure clustering on DLT tables

[Chris_N]

Hi Team

I have a DLT pipeline with `cluster_by` property configured for all my tables. The code looks something like below:

@Dlt.table( name="flows", cluster_by=["from"] ) def flows(): <LOGIC>

It was all working fine and in couple of days, the queries were taking forever and when I checked my dlt tables. I couldn't find any cluster properties configured. I tried setting 'cluster_by_auto=True' and it was properly configured but the cluster columns are not taken into consideration. 

Is this some bug with the latest release or is there a way to solve this?

Thanks in advance

[NandiniN]

Hi @Chris_N , 

 You have mentioned - "I couldn't find any cluster properties configured."

If they existed and were changed, you can use the delta history command to check if someone changed on the clustering information. 

It is possible there were changes in the data volume/configs that could have led to the change in performance.

On the next statement of yours - " I tried setting 'cluster_by_auto=True' and it was properly configured but the cluster columns are not taken into consideration." - How do you determine this that, the cluster columns are not considered?

After liquid clustering is enabled, run OPTIMIZE jobs as usual to incrementally cluster data. See How to trigger clustering.

Liquid clustering is incremental, meaning that data is only rewritten as necessary to accommodate data that needs to be clustered. Data files with clustering keys that do not match the data to be clustered are not rewritten.

Please let me know of additional informations, so that I can suggest further. Additionally, if you have a support subscription, a support ticket can also be raised to investigate this.

Thanks & Regards,

Nandini