Delete a directory in DBFS recursively from Azure

Yannic
New Contributor

‎06-18-2025 04:22 AM

I have an Azure storage mounted to DBFS. I want to delete a directory inside recursively. I tried both,

dbutils.fs.rm(f"/mnt/data/to/delete", True)

and

%fs rm -r /mnt/data/to/delete

 In both cases I get the following exception:

AzureException: hadoop_azure_shaded.com.microsoft.azure.storage.StorageException: This operation is not permitted on a non-empty directory.
Caused by: StorageException: This operation is not permitted on a non-empty directory.

 How can I delete a directory recursively?

Thanks!

0 Kudos

lingareddy_Alva
Esteemed Contributor

‎06-18-2025 10:01 AM - edited ‎06-18-2025 10:02 AM

Hi @Yannic 

Azure Blob Storage doesn't have true directories - it simulates them through blob naming conventions,
which can cause issues with recursive deletion operations.

Try below one. Delete Files First, Then Directory

def delete_directory_recursive(path):
    try:
        # First, list all files and subdirectories
        files = dbutils.fs.ls(path)
        
        for file_info in files:
            if file_info.isDir():
                # Recursively delete subdirectories
                delete_directory_recursive(file_info.path)
            else:
                # Delete individual files
                dbutils.fs.rm(file_info.path)
        
        # Finally, delete the empty directory
        dbutils.fs.rm(path)
        print(f"Successfully deleted: {path}")
        
    except Exception as e:
        print(f"Error deleting {path}: {str(e)}")

# Usage
delete_directory_recursive("/mnt/data/to/delete")

 

 

LR
0 Kudos