topic Re: remove empty folders with pyspark in Data Engineering

remove empty folders with pyspark

Nathant93 — Tue, 09 Jul 2024 15:13:47 GMT

Hi,

I am trying to search a mnt point for any empty folders and remove them. Does anyone know of a way to do this? I have tried dbutils.fs.walk but this does not seem to work.

Thanks

Re: remove empty folders with pyspark

Nathant93 — Mon, 15 Jul 2024 12:41:30 GMT

Unfortunately this says that every folder in my mnt point has a size of 0. I have folders that have folders in which then might contain a metadata file for a streaming checkpoint.

Re: remove empty folders with pyspark

MathieuDB — Thu, 16 Jan 2025 14:26:39 GMT

Hello @Nathant93,

You could use dbutils.fs.ls and iterate on all the directories found to accomplish this task.

Something like this:

def find_empty_dirs(path): directories = dbutils.fs.ls(path) for directory in directories: if directory.isDir(): find_empty_dirs(directory.path) contents = dbutils.fs.ls(directory.path) if len(contents) == 0: # Logic dbutils.fs.rm(directory.path, recurse=True) print(f"Removed empty directory: {directory.path}") find_empty_dirs("dbfs:/mnt/your_mount_point")