AbhilashNagilla
Databricks Employee
Databricks Employee

I have not found a supported way to make Unity Catalog release that path early, but you can get Auto Loader running again today without it.

The reservation belongs to the managed table your access test created. Dropping it starts a recovery window rather than deleting it: per Object storage lifecycle in Unity Catalog, "For 7 days after the delete, Unity Catalog retains the soft-deleted data so that you can recover the object", and during that window "Unity Catalog retains the dropped object's metadata". That record is what your three attempts could not reach. Deleting __unitystorage removed files rather than metadata, the "1 dependent managed tables" warning was the record itself, and re-adding the external location put it back over a path that is still reserved. UNDROP is not open to you either, since the same page lists catalogs as "Cannot be recovered after delete", leaving nothing to scope it to.

To get unblocked now, point Auto Loader at a subdirectory such as /incoming/ and land the source files there instead of the container root. The path in your error is the Auto Loader input path, and your external location on the container root already authorizes everything beneath it, so you should not need to touch the external location itself. Specify a managed storage location documents the reserved subtree as <storage-root>/__unitystorage/catalogs/<uuid>, and a sibling directory does not overlap that.

I tried this on an S3 bucket root with managed storage underneath, external location left on the root: reading a sibling directory passed the path check, while a path inside __unitystorage returned your error. That catalog still existed, so treat it as the right shape rather than proof for a dropped one.

Worth ruling out a second live registration as well. Run this as a metastore admin:

SELECT table_catalog, table_schema, table_name, storage_path
FROM system.information_schema.tables
WHERE table_type = 'MANAGED'
  AND storage_path LIKE 'abfss://container@mystorageaccount.dfs.core.windows.net%';

Soft-deleted tables never appear there, and results are filtered to objects you have privileges on, so read a clean result narrowly.

Worth avoiding next time: Unity Catalog best practices advises against using "a single storage account and container for managed storage and other external locations", which is the shape that produced this.