cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Transition from partitioned table to Liquid clustered table

mjedy78
New Contributor II

Hi all,
I have a table called classes, which is already partitioned on three different columns. I want to create a Liquid Clustered Table, but as far as I understand from the documentationโ€”and from Dany Lee and his teamโ€”it was not possible as of 2024 YouTube Video.

I believe there is still no smooth transition. So, I want to recreate the table, but is the only way to do this by creating a new table, deleting the old one, and moving everything from new_table to the old tableโ€™s location?

 

%sql
-- Step1
CREATE TABLE IF NOT EXISTS class_clustered
CLUSTER BY (id, family, result_date)
LOCATION 'my_new_table_location'
AS
SELECT *
FROM class(the main partitioned table);

-- Step2
DROP class

-- Step3
Remove location in dbfs

-- Step4
Move everything from location my_new_table_location --> old location(for class partitioned)

 

Can I overwrite it or do something similar without too much effort? ๐Ÿ˜ƒ There are more than 20 tables that need to be converted to Liquid Clustered Tables, so Iโ€™m looking for the best approach.

 

Join us on Thursday, December 7 at 10AM PST for an enlightening session on Delta Lake's Liquid Clustering, a transformative approach in data management and optimization with Vรญtor Teixeira, Senior Data Engineer at Veeva Systems. Liquid Clustering is Delta Lake's answer to the complex challenges of
3 REPLIES 3

MariuszK
Contributor III

You can enable liquid clustering on an existing table:

ALTER TABLE <table_name>
CLUSTER BY (<clustering_columns>)

You need to run OPTIMIZE FULL to force reclustering.

mjedy78
New Contributor II

@MariuszK It is for normal tables and will not work with the partitioned ones 

mjedy78_0-1740038996897.png

 

 

MariuszK
Contributor III

I'm sorry, I missed that your tables are partitioned, in this case easiest  way will be to recreate it.

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now