Delete Partition Folders
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-26-2024 04:35 AM
Hello team, as DataBricks moved away from hive-style partitioning, we can see some 2-letter partition folders created. And I have observed that the vacuum doesn't delete these folders (even though they are empty). Is there any way to delete those using built-in features like REORG?
Of course, we can do it by looping over folders/subfolders, checking if they are empty, and then deleting but I am afraid it might break some logs, etc.
I would appreciate your help. If anyone has a better idea, please do share.
- Labels:
-
Delta Lake
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-26-2024 06:27 AM
Hello @shadowinc,
VACUUM is used to clean up unused and stale data files that are no longer referenced by a Delta table and are older than a specified retention period (default is 7 days). It does not remove empty directories. I think manual cleanup will be the way to go. I will check if any other workaround and let you know.

