I have a catalog with multiple schemas, tables and volumes. Each volume contains two folders with specific names that I cannot change.
When I drop the whole catalog (CASCADE) and create catalog with the same name, with the same schemas, tables and volumes, inside volumes I cannot create those folders neither spark nor dbutils. It works via UI though... I get error:
Failed to acquire a SAS token for create-directory on /my_catalog/__unitystorage/catalogs/35a66a07-e1d4-4f0a-931a-8ba1d0f4578e/volumes/4ae1832e-f9c0-470d-9c06-80fc7a290849/src due to java.util.concurrent.ExecutionException: com.databricks.sql.managedcatalog.UnityCatalogServiceException: [RequestId=8e073a68-fd74-43c7-a57b-a0879480d0c2 ErrorClass=SCHEMA_DOES_NOT_EXIST.RESOURCE_DOES_NOT_EXIST] Schema '1db0471a-0e8d-450c-bb3b-bfd216ef26ad' does not exist.
those IDs refers to the previously dropped catalog, not the current one.
After 24 hours the same command succeeds. Is there any metadata caching? Can I clean it manually?