topic Re: How does running VACUUM on Delta Lake tables effect read/write performance? in Data Engineering

How does running VACUUM on Delta Lake tables effect read/write performance?

User16783853906 — Thu, 10 Jun 2021 21:47:11 GMT

If I don't run VACUUM on a Delta Lake table, will that make my read performance slower?

Re: How does running VACUUM on Delta Lake tables effect read/write performance?

sajith_appukutt — Thu, 17 Jun 2021 22:39:29 GMT

VACUUM does not have a direct impact on read/write performance since it only remove files no longer referenced by a Delta table ( unless your data volume is so high that you are hitting the read limits of underlying S3/GCS/ADLS buckets ) . It would make sense to run it as a separate job scheduled daily and potentially using sport instances

Re: How does running VACUUM on Delta Lake tables effect read/write performance?

User16783853906 — Wed, 23 Jun 2021 21:24:26 GMT

VACUUM has no effect on read/write performance to that table. Never running VACUUM on a table will not make read/write performance to a Delta Lake table any slower.

If you run VACUUM very infrequently, your VACUUM runtimes themselves may be pretty high, so it is suggested to run VACUUM somewhat regularly. How often you should run VACUUM depends on your storage costs.