Delete Partition Folders

shadowinc — Tue, 26 Nov 2024 12:35:14 GMT

Hello team, as DataBricks moved away from hive-style partitioning, we can see some 2-letter partition folders created. And I have observed that the vacuum doesn't delete these folders (even though they are empty). Is there any way to delete those using built-in features like REORG?

Of course, we can do it by looping over folders/subfolders, checking if they are empty, and then deleting but I am afraid it might break some logs, etc.

I would appreciate your help. If anyone has a better idea, please do share.

Re: Delete Partition Folders

Alberto_Umana — Tue, 26 Nov 2024 14:27:57 GMT

Hello @shadowinc,

VACUUM is used to clean up unused and stale data files that are no longer referenced by a Delta table and are older than a specified retention period (default is 7 days). It does not remove empty directories. I think manual cleanup will be the way to go. I will check if any other workaround and let you know.

topic Re: Delete Partition Folders in Data Engineering

Delete Partition Folders

Re: Delete Partition Folders