Liquid Clustering With more than 4 columns

[Erfan]

Hi there,

I’m trying to join a small table (a few million records) with a much larger table (around 1 TB in size, containing a few billion records).

The small table isn’t quite small enough to use Broadcast. Additionally, our join clause involves more than four columns. I attempted to enable Liquid Clustering on the large table, but it only supports up to four columns. I experimented with different combinations of four-column sets for Liquid Clustering, but none of them reduced the join time.

Do you have any recommendations for optimizing a query on a table with Liquid Clustering when the join criteria involve more than four columns?

[filipniziol · Accepted solution]

Hi @Erfan ,

What you can do is to create an additional column that concatenates the values of multiple columns and then apply Liquid Clustering on that new column.

[Erfan]

Hi @filipniziol ,

Good idea. I'll try it and will come back with the result. Thanks! 

[Erfan]

Unfortunatelly, since I am not the owner of the data, I am not allowed to add additional column. So I can't test it. But I guess your idead