topic Issue with Auto Liquid clustering in Data Engineering

Issue with Auto Liquid clustering

RevanthV — Sat, 13 Sep 2025 07:58:23 GMT

I have written data to a table using clusterByAuto set to true
But the clustering keys are not selected automatically when i do a desc detail on the table.Screenshot below

Why are clustering columns not being selected automatically?

Repro steps:

Create a dataframe
Write to a delta table using clusterByAuto set to true
Then run desc detail on teh table .You will not see any clustering columns selected

Re: Issue with Auto Liquid clustering

K_Anudeep — Sat, 13 Sep 2025 08:42:55 GMT

Hello Revanth,

It could be that the

The table is too small to benefit from liquid clustering, or it has a good clustering scheme. Could you tell me the size of the table?
Also, if the table is not being frequently queried on a column or set of columns,ie.., there are not enough scans on the table, so it wouldn't benefit from clustering
Also, Auto Liquid works with PO in the background, so based on some internal thresholds, the clustering columns are selected, and we as Cx need not worry about that

So once you have frequent scans on the table columns, you will see clustering columns being selected automatically.

You can go through the doc for better understanding: https://docs.databricks.com/aws/en/delta/clustering#:~:text=If%20a%20key%20was%20not%20selected%20by%20automatic%20liquid%20clustering%2C%20the%20reason%20can%20be%3A

Let me know if you have any further questions.

Re: Issue with Auto Liquid clustering

szymon_dybczak — Sat, 13 Sep 2025 09:04:52 GMT

Hi @RevanthV ,

As @K_Anudeep correctly suggested it could be the case that your table is to small to benefit from liquid clustering.
Another possibility it that you're using runtime lower than 15.4 LTS.

Re: Issue with Auto Liquid clustering

RevanthV — Sat, 13 Sep 2025 11:51:51 GMT

Thanks a lot @K_Anudeep , My table is still small and i guess that was the reason , have now written around 1million records and have been just running frequent scans on a column since last one hour and now i can see the same column selected as a clustering column.

Thanks a lot for your help