Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
I agree with you @Werner Stinckensโ . I will always do optimize to try to compact the files more (default size is 1 gb) and then run vacuum. Vacuum will remove the old files that are no longer needed or have been marked as obsoleted by the optimize command.
In my opinion as Vacuum remove old files so is better to do Vacuum first so there will be less files to optimize but maybe in some cases it will be faster to delete optimized already files (so optimize first) โ๏ธ
I optimize first as delta lake knows which files are relevant for the optimize. Like that I have my optimized data available faster. Then a vacuum. Seemed logical to me, but I might be wrong. Never actually thought about it
I agree with you @Werner Stinckensโ . I will always do optimize to try to compact the files more (default size is 1 gb) and then run vacuum. Vacuum will remove the old files that are no longer needed or have been marked as obsoleted by the optimize command.
Does it help or make sense to add Re-org then Optimize - Vacuum every week?
Reorganize a Delta Lake table by rewriting files to purge soft-deleted data, such as the column data dropped by ALTER TABLE DROP COLUMN. Does it mean, it only touches the delta logs for deleted columns or data and not the actual data?
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
