Delete a directory in DBFS recursively from Azure

Yannic — Wed, 18 Jun 2025 11:22:27 GMT

I have an Azure storage mounted to DBFS. I want to delete a directory inside recursively. I tried both,

dbutils.fs.rm(f"/mnt/data/to/delete", True)

and

%fs rm -r /mnt/data/to/delete

In both cases I get the following exception:

AzureException: hadoop_azure_shaded.com.microsoft.azure.storage.StorageException: This operation is not permitted on a non-empty directory. Caused by: StorageException: This operation is not permitted on a non-empty directory.

How can I delete a directory recursively?

Thanks!

Re: Delete a directory in DBFS recursively from Azure

lingareddy_Alva — Wed, 18 Jun 2025 17:02:35 GMT

Hi @Yannic

Azure Blob Storage doesn't have true directories - it simulates them through blob naming conventions,
which can cause issues with recursive deletion operations.

Try below one. Delete Files First, Then Directory

def delete_directory_recursive(path): try: # First, list all files and subdirectories files = dbutils.fs.ls(path) for file_info in files: if file_info.isDir(): # Recursively delete subdirectories delete_directory_recursive(file_info.path) else: # Delete individual files dbutils.fs.rm(file_info.path) # Finally, delete the empty directory dbutils.fs.rm(path) print(f"Successfully deleted: {path}") except Exception as e: print(f"Error deleting {path}: {str(e)}") # Usage delete_directory_recursive("/mnt/data/to/delete")

topic Re: Delete a directory in DBFS recursively from Azure in Data Engineering

Delete a directory in DBFS recursively from Azure

Re: Delete a directory in DBFS recursively from Azure