remove empty folders with pyspark

[Nathant93]

Hi,

I am trying to search a mnt point for any empty folders and remove them. Does anyone know of a way to do this? I have tried dbutils.fs.walk but this does not seem to work.

Thanks

[Kaniz_Fatma]

Hi @Nathant93, 

To find and remove empty folders in a mount point using PySpark, you can follow these steps:

1. List all folders in the mount point
You can use the `dbutils.fs.ls()` function to list all the folders in the mount point:

folders = dbutils.fs.ls("/mnt/myMountPoint")

This will give you a list of `FileInfo` objects, each representing a folder or file in the mount point.

2. Filter for empty folders
To find the empty folders, you can filter the list of `FileInfo` objects to only include those that represent folders and have a size of 0 bytes:

empty_folders = [f for f in folders if f.isDir and f.size == 0]

3. Remove the empty folders
Once you have the list of empty folders, you can use the `dbutils.fs.rm()` function to delete them:

for folder in empty_folders:
dbutils.fs.rm(folder.path, True)

The `True` argument tells `dbutils.fs.rm()` to recursively delete the folder and its contents.

Here's the complete code:

# List all folders in the mount point
folders = dbutils.fs.ls("/mnt/myMountPoint")

# Filter for empty folders
empty_folders = [f for f in folders if f.isDir and f.size == 0]

# Remove the empty folders
for folder in empty_folders:
dbutils.fs.rm(folder.path, True)

This will find all the empty folders in the specified mount point and delete them. Make sure to replace `/mnt/myMountPoint` with the actual mount point you want to search.

[Nathant93]

Unfortunately this says that every folder in my mnt point has a size of 0. I have folders that have folders in which then might contain a metadata file for a streaming checkpoint.