cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Liquid Clustering With more than 4 columns

Erfan
New Contributor III

Hi there,

Iโ€™m trying to join a small table (a few million records) with a much larger table (around 1 TB in size, containing a few billion records).

The small table isnโ€™t quite small enough to use Broadcast. Additionally, our join clause involves more than four columns. I attempted to enable Liquid Clustering on the large table, but it only supports up to four columns. I experimented with different combinations of four-column sets for Liquid Clustering, but none of them reduced the join time.

Do you have any recommendations for optimizing a query on a table with Liquid Clustering when the join criteria involve more than four columns?

 

1 ACCEPTED SOLUTION

Accepted Solutions

filipniziol
Contributor III

Hi @Erfan ,

What you can do is to create an additional column that concatenates the values of multiple columns and then apply Liquid Clustering on that new column.

View solution in original post

3 REPLIES 3

filipniziol
Contributor III

Hi @Erfan ,

What you can do is to create an additional column that concatenates the values of multiple columns and then apply Liquid Clustering on that new column.

Erfan
New Contributor III

Hi @filipniziol ,

Good idea. I'll try it and will come back with the result. Thanks! 

Erfan
New Contributor III

Unfortunatelly, since I am not the owner of the data, I am not allowed to add additional column. So I can't test it. But I guess your idead

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group