implementing liquid clustering for DataFrames directly

luriveros
New Contributor

‎12-19-2023 02:01 PM

 

Hi !! I have a question is it possible to implementing liquid clustering for DataFrames directly saved to delta files (df.write.format("delta").save("path")), The conventional approach involving table creation
0 Kudos

brockb
Databricks Employee
Databricks Employee

‎02-05-2024 08:10 PM

Hi,
Hopefully this question is related to testing and any production data would get persisted to a table but one example is:

df = (
spark.range(10)
.write
.format("delta")
.mode("append")
.save("file:/tmp/data")
)

ALTER TABLE delta.`file:/tmp/data` CLUSTER BY (id);

DESC DETAIL delta.`file:/tmp/data`
OPTIMIZE delta.`file:/tmp/data`;
 
Thanks.
0 Kudos