topic Re: Optimize and Vacuum - which is the best order of operations? in Data Engineering

Optimize and Vacuum - which is the best order of operations?

Hubert-Dudek — Wed, 03 Nov 2021 16:26:02 GMT

Optimize -> Vacuum

Vacuum -> Optimize

Re: Optimize and Vacuum - which is the best order of operations?

Hubert-Dudek — Wed, 03 Nov 2021 16:26:42 GMT

I am asking some questions as nobody is writing today 🙂

Re: Optimize and Vacuum - which is the best order of operations?

Piper_Wilson — Wed, 03 Nov 2021 16:28:57 GMT

Thank you!

Re: Optimize and Vacuum - which is the best order of operations?

Hubert-Dudek — Wed, 03 Nov 2021 16:30:47 GMT

In my opinion as Vacuum remove old files so is better to do Vacuum first so there will be less files to optimize but maybe in some cases it will be faster to delete optimized already files (so optimize first) ❔️

Re: Optimize and Vacuum - which is the best order of operations?

-werners- — Thu, 04 Nov 2021 12:17:41 GMT

I optimize first as delta lake knows which files are relevant for the optimize. Like that I have my optimized data available faster. Then a vacuum. Seemed logical to me, but I might be wrong. Never actually thought about it

Re: Optimize and Vacuum - which is the best order of operations?

jose_gonzalez — Fri, 05 Nov 2021 00:30:20 GMT

I agree with you @Werner Stinckens . I will always do optimize to try to compact the files more (default size is 1 gb) and then run vacuum. Vacuum will remove the old files that are no longer needed or have been marked as obsoleted by the optimize command.

Re: Optimize and Vacuum - which is the best order of operations?

shadowinc — Tue, 26 Nov 2024 12:04:15 GMT

What about ReOrg delta table https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/delta-reorg-table

Does it help or make sense to add Re-org then Optimize - Vacuum every week?

Reorganize a Delta Lake table by rewriting files to purge soft-deleted data, such as the column data dropped by ALTER TABLE DROP COLUMN.
Does it mean, it only touches the delta logs for deleted columns or data and not the actual data?