Disabled S3 bucket version, Next steps?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-07-2024 06:24 AM - edited 02-07-2024 06:27 AM
We have been using Databricks for some time and didn't knew that S3 bucket versioning was not recommended. We have disabled it now. What are the next steps that we need to take to clean up the data, should we create a lifecycle rule to delete older versions manually or there is a way databricks handles that itself.
PS: We are running vaccum on that tables in our catalogue, but need general guidance on how to manage this situation to save on storage costs without impacting any current flow.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-07-2024 11:46 PM
@prashantjjain33 if you had enabled S3 Versioning, having the S3 Lifecycle is one approach that you can consider.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-08-2024 01:03 AM
Adding lifecycle policies to databricks s3 bucket does not impact existing data, is that correct @Yeshwanth .

