topic VACUUM during read/write in Data Engineering

VACUUM during read/write

User16783853906 — Thu, 10 Jun 2021 21:49:06 GMT

Is it safe to run VACUUM on a Delta Lake table while data is being added to it at the same time? Will it impact the job result/performance?

Re: VACUUM during read/write

sajith_appukutt — Thu, 17 Jun 2021 19:06:56 GMT

If you are running VACUUM with a very short retention interval, old snapshots and uncommitted files can still be in use by concurrent readers or writers to the table and this could result in concurrent readers to fail or tables getting corrupted

Delta Lake has a safety check spark.databricks.delta.retentionDurationCheck.enabled set to true by default to prevent you from running a dangerous vacuum command .

More details could be found here. https://docs.databricks.com/delta/delta-utility.html#remove-files-no-longer-referenced-by-a-delta-table

From a performance point of you, to ensure that it is not affecting the other running jobs, you could run it in a job cluster

Re: VACUUM during read/write

User16783853906 — Wed, 23 Jun 2021 21:26:03 GMT

In the vast majority of cases, yes, it is safe to run VACUUM while data is concurrently being appended or updated to the same table. This is because VACUUM deletes data files no longer referenced by a Delta table's transaction log and does not effect the current snapshot that data is being operated on by other processes.

However, you will want to be careful if you're specifying a shorter-than-default retention period or if you have streams that run very infrequently.