Delete Partition Folders

shadowinc
New Contributor III

Hello team, as DataBricks moved away from hive-style partitioning, we can see some 2-letter partition folders created. And I have observed that the vacuum doesn't delete these folders (even though they are empty). Is there any way to delete those using built-in features like REORG?

Of course, we can do it by looping over folders/subfolders, checking if they are empty, and then deleting but I am afraid it might break some logs, etc.

I would appreciate your help. If anyone has a better idea, please do share.

Alberto_Umana
Databricks Employee
Databricks Employee

Hello @shadowinc,

VACUUM is used to clean up unused and stale data files that are no longer referenced by a Delta table and are older than a specified retention period (default is 7 days). It does not remove empty directories. I think manual cleanup will be the way to go. I will check if any other workaround and let you know.