cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

implementing liquid clustering for DataFrames directly

luriveros
New Contributor

 

Hi !! I have a question is it possible to implementing liquid clustering for DataFrames directly saved to delta files (df.write.format("delta").save("path")), The conventional approach involving table creation
1 REPLY 1

brockb
Valued Contributor
Valued Contributor

Hi,
Hopefully this question is related to testing and any production data would get persisted to a table but one example is:

df = (
spark.range(10)
.write
.format("delta")
.mode("append")
.save("file:/tmp/data")
)

ALTER TABLE delta.`file:/tmp/data` CLUSTER BY (id);

DESC DETAIL delta.`file:/tmp/data`
OPTIMIZE delta.`file:/tmp/data`;
 
Thanks.