topic Table Properties different for liquid clustering with Databricks version. in Data Engineering

Table Properties different for liquid clustering with Databricks version.

NhanNguyen — Fri, 13 Dec 2024 03:45:53 GMT

Dear all,

Today, I tried the liquid clustering in Databricks, but after running it with two Databricks engine version, it showed different properties in the catalog explorer.

1. Run with DBR version 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12) it will show:

clusteringColumns=[["col1"],["col2"]]

2. Run with DBR version 13.3 LTS (includes Apache Spark 3.4.1, Scala 2.12) it will show:

delta.liquid.clusteringColumns=col1,col2

I have a question: Why do we have these differences, and were both case tables successfully enabled for liquid clustering?

Thanks guys!

Re: Table Properties different for liquid clustering with Databricks version.

holly — Fri, 13 Dec 2024 11:18:16 GMT

Hi there, can I get a bit more detail?

Is this the same table that is showing differently depending on what you used to query it last? Or is this two separate tables created with different runtimes?
Are there any differences in the metadata files that are stored alongside the parquet files?
Are the tables the same delta versions?

I have a suspicion that the later version has an updated way of representing the columns

Re: Table Properties different for liquid clustering with Databricks version.

VZLA — Fri, 13 Dec 2024 15:36:30 GMT

Correct, like @holly rightly said this is just an updated way of representing the columns in a more structured or updated manner, it may also be matching a new value type. In both cases the table property is reflecting that LC was enabled. Our suggestion is to go with DBR 15 as there are improvements and validations on top worth leveraging.