- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-17-2021 12:06 PM
If you are running VACUUM with a very short retention interval, old snapshots and uncommitted files can still be in use by concurrent readers or writers to the table and this could result in concurrent readers to fail or tables getting corrupted
Delta Lake has a safety check spark.databricks.delta.retentionDurationCheck.enabled set to true by default to prevent you from running a dangerous vacuum command .
More details could be found here. https://docs.databricks.com/delta/delta-utility.html#remove-files-no-longer-referenced-by-a-delta-ta...
From a performance point of you, to ensure that it is not affecting the other running jobs, you could run it in a job cluster