I've tried to collect the paths of the inner partitions recursively, list the paths, and delete them in parallel. (https://kb.databricks.com/data/list-delete-files-faster) Here is the online example. I get it to run but the delete is so slow that it will take months and months even when I scale up compute. I tried your code but results are the same dbutils.fs.ls(path) exhausts memory.

It is hard to believe this is the easiest way to delete a massive amount of messy data from Azure storage in the Databricks managed resource. In regular Azure storage not managed by databricks, this would be extremely easy to delete directly in Azure portal.

If there isn't an answer then we will literally have to migrate to a completely new separate databricks resource, then delete this whole databricks resource in order to delete the storage and eliminate the unnecessary costs. We have legacy processes that still run on this resource where this would take a lot of work. This is a poor solution path to just delete data.