I need to delete 50TB of data out of dfbs storage. It is overpartitioned and dbutils does not work. Also, limiting partition size and iterating over data to delete doesn't work. Azure locks access from storage from the resource group permissions and even though I'm the owner and admin of the Azure subscription I still can't grant myself access.

I've isolated the problem to a single hive_metastore catalog that was deleted yet the directory and data for two "tables" still remain. It has been over 30 days and was not cleaned out even though it is deleted. This extra data has increased our storage costs significantly.

https://community.databricks.com/t5/data-engineering/how-do-i-delete-files-from-the-dbfs/td-p/29185 doesn't work

https://stackoverflow.com/questions/54091812/remove-files-from-directory-after-uploading-in-databric... doesn't work

https://stackoverflow.com/questions/54081662/how-to-delete-all-files-from-folder-with-databricks-dbu... doesn't work

https://docs.databricks.com/api/workspace/dbfs/delete doesn't work and it says "when you delete a large number of files, the delete operation is done in increments." 

https://kb.databricks.com/data/list-delete-files-faster works but is so slow it would take months and months to run through this data that is extremely overpartitioned.

In regular Azure blob storage that isn't auto-created and managed by databricks, it is very easy to delete data. Click on the container or directory or file then click delete and it is gone.

How do I just delete a massive amount of data in databricks storage?

Thanks in advance.