VACUUM during read/write

[User16783853906]

Is it safe to run VACUUM on a Delta Lake table while data is being added to it at the same time?  Will it impact the job result/performance?

[sajith_appukutt]

If you are running VACUUM with a very short retention interval, old snapshots and uncommitted files can still be in use by concurrent readers or writers to the table and this could result in concurrent readers to fail or tables getting corrupted

Delta Lake has a safety check spark.databricks.delta.retentionDurationCheck.enabled set to true by default to prevent you from running a dangerous  vacuum command .

More details could be found here. https://docs.databricks.com/delta/delta-utility.html#remove-files-no-longer-referenced-by-a-delta-ta...

From a performance point of you, to ensure that it is not affecting the other running jobs, you could run it in a job cluster

[User16783853906]

In the vast majority of cases, yes, it is safe to run VACUUM while data is concurrently being appended or updated to the same table. This is because VACUUM deletes data files no longer referenced by a Delta table's transaction log and does not effect the current snapshot that data is being operated on by other processes.

However, you will want to be careful if you're specifying a shorter-than-default retention period or if you have streams that run very infrequently.