delete non empty folders from workspace

[Cloudtek]

Looking for a solution to delete 1000's of non-empty folders that I created wrongly while migration. (looking for solution that delete thoudands of folders created with below name)

Folders have a path like "shared/db/..../..../..../yippy.dbc"

[Kaniz_Fatma]

Hi @Cloudtek, To delete thousands of non-empty folders in Databricks, you can use the following approaches:

1. Databricks Notebook Command:

   • In a Databricks notebook, execute the following Python commands:

     dbutils.fs.refreshMounts()
     dbutils.fs.rm("/mnt/my_folder", recurse=True)
     

     This will refresh the state of the mounts in your workspace and then attempt to delete the folder re...¹.

2. Parallel Deletion with Apache Spark™:

   • If you have a large number of folders to delete, consider using an Apache Spark™ job to list and delete files faster. Here’s how:
     • Collect the paths of the inner partitions recursively.
     • List the paths.
     • Delete them in parallel².

3. Databricks CLI:

   • Use the Databricks CLI to delete folders. For example:

     databricks workspace delete -r dbfs:/foobar
     

     Replace dbfs:/foobar with the actual path of the folder you want to delete³.

Remember to replace /mnt/my_folder and dbfs:/foobar with the actual paths of the folders you need to delete. Be cautious when deleting data to avoid accidental loss. 🚀