INVALID_PARAMETER_VALUE.LOCATION_OVERLAP after catalog deleted

[emorgoch]

I've got an ADLS Gen 2 container that has cloud files that I want to ingest with AutoLoader. The files are saved into the root of the container.

I created an external location for that container to be able to access it through managed credentials. However, in my script used to set it up, I also accidentally created a catalog, schema, and table as a test to validate access. No problem, I thought. I deleted the catalog within databricks, and then deleted __unitystorage directory & files.

However, now when I try to have autoloader access the files in that container I'm getting the following error:

AnalysisException: [RequestId=8f219926-d946-4601-a683-18495a6ded8f ErrorClass=INVALID_PARAMETER_VALUE.LOCATION_OVERLAP] Input path url 'abfss://container@mystorageaccount.dfs.core.windows.net/' overlaps with managed storage within 'CheckPathAccess' call.

If I try and delete the external location from Unity Catalog, Databricks prompts me that it can't delete it due to the location having 1 dependent managed tables.

I used the Force delete option, and then re-added the external location, and I'm still getting LOCATION_OVERLAP exception when trying to read the files with AutoLoader.

I tried using SHOW DROPPED TABLES, but that doesn't work cause as the catalog was dropped first.

Does anyone have any suggestions on how I can get UC to recognize that this container root is no longer being used for any managed resources?

[GabFernandes · Accepted solution]

Hi @emorgoch 

This is a known pain point. Here's what's happening and how to resolve it:

---

Root Cause

When you created the catalog/schema/table using that container as managed storage, Unity Catalog registered that path internally as a managed storage location. Even after you:

• Dropped the catalog
• Deleted the __unitystorage directory from ADLS
• Force-deleted and re-created the external location

...the metastore still retains metadata about that path being associated with managed storage. This metadata persists for at least 7 days (the soft-delete recovery window), and in some cases can become "orphaned" if the catalog was dropped before the tables underneath were fully purged.

The LOCATION_OVERLAP error occurs because UC prevents external locations (or operations like AutoLoader READ FILES) from accessing paths that overlap with known managed storage locations — even if the actual data is gone.

---

Solutions (in order of preference)

1. Wait for the 7-day purge window

After dropping a catalog with managed tables, UC retains soft-delete metadata for 7 days. After the purge completes (~7 days + up to 48h for final cleanup), the managed storage reference should be released.

Check: How long ago did you drop the catalog? If less than 7-9 days, wait it out.

2. Contact Databricks Support (recommended if >9 days)

If the 7-day window has passed and you're still seeing LOCATION_OVERLAP, the managed storage reference is likely orphaned in the metastore metadata. This can happen when:

• The catalog was dropped with CASCADE but the purge didn't fully complete
• The __unitystorage directory was manually deleted from storage before UC could clean up
• The external location was force-deleted while managed table metadata still existed

Databricks Support can manually remove the orphaned managed storage reference from your metastore's internal metadata. Provide them:

• The external location name: planview-adaptivework_psinkext-udp_extloc
• The storage URL (the abfss://...dfs.core.windows.net/ path)
• The error message with RequestId
• Confirmation that the catalog and all its contents are intentionally deleted

3. Workaround: Use a subdirectory-scoped external location

If you need to unblock immediately while waiting for Support:

Instead of pointing the external location at the container root:

abfss://container@account.dfs.core.windows.net/

Create the external location pointing to a subdirectory where your files actually live:

abfss://container@account.dfs.core.windows.net/data/

Or move your landing files into a subfolder. The overlap check is path-based — a subdirectory that doesn't collide with the old __unitystorage path (which was at the root level) should work.

4. Verify there are no hidden references

Run these queries to check for any lingering references in your metastore:

-- Check if any catalogs still reference this storage location
SELECT * FROM system.information_schema.catalog_tags 
WHERE catalog_name LIKE '%<your_catalog_name>%';

-- Check all external locations for overlapping paths
SHOW EXTERNAL LOCATIONS;

-- Look for managed storage locations at the metastore level
DESCRIBE METASTORE;

Also verify with the Unity Catalog REST API:

GET /api/2.1/unity-catalog/external-locations

to see if there's a hidden managed storage location you're not seeing in the UI.

---

Prevention for the future

1. Never use a container root as managed storage if you also plan to use it for external data/AutoLoader. Use a dedicated container or a dedicated subdirectory for managed storage.
2. Don't manually delete __unitystorage from storage — let UC manage the lifecycle. Manual deletion can orphan metadata.
3. Drop tables/schemas before dropping the catalog (rather than CASCADE) if you want finer control over cleanup.
4. Separate concerns: Use one container/path for managed tables and a different one for landing files/external data.

---

TL;DR

The LOCATION_OVERLAP persists because UC's metastore still has an internal reference marking that container root as "managed storage" — even after you deleted everything visually. The force-delete of the external location + physical deletion of __unitystorage didn't clear the metastore's managed-storage registry.

Quick fix: Point external location to a subdirectory instead of root. Permanent fix: Wait 7-9 days for purge, or open a Databricks Support ticket to manually remove the orphaned managed storage reference.

[emorgoch]

Thanks @GabFernandes, It wasn't intended to co-habitate the managed resources with the ingestion files, just an error in the setup that lead to it. I think the 7 day timeframe expires tomorrow, so hopefully that will resolve it, else I will get a hold of databricks support to see if they can clean it up.

[GabFernandes]

YW!

Amazing. Let's get back here tomorrow so we can see if it's works 🙂

Thanks!!

[AbhilashNagilla]

I have not found a supported way to make Unity Catalog release that path early, but you can get Auto Loader running again today without it.

The reservation belongs to the managed table your access test created. Dropping it starts a recovery window rather than deleting it: per Object storage lifecycle in Unity Catalog, "For 7 days after the delete, Unity Catalog retains the soft-deleted data so that you can recover the object", and during that window "Unity Catalog retains the dropped object's metadata". That record is what your three attempts could not reach. Deleting __unitystorage removed files rather than metadata, the "1 dependent managed tables" warning was the record itself, and re-adding the external location put it back over a path that is still reserved. UNDROP is not open to you either, since the same page lists catalogs as "Cannot be recovered after delete", leaving nothing to scope it to.

To get unblocked now, point Auto Loader at a subdirectory such as /incoming/ and land the source files there instead of the container root. The path in your error is the Auto Loader input path, and your external location on the container root already authorizes everything beneath it, so you should not need to touch the external location itself. Specify a managed storage location documents the reserved subtree as <storage-root>/__unitystorage/catalogs/<uuid>, and a sibling directory does not overlap that.

I tried this on an S3 bucket root with managed storage underneath, external location left on the root: reading a sibling directory passed the path check, while a path inside __unitystorage returned your error. That catalog still existed, so treat it as the right shape rather than proof for a dropped one.

Worth ruling out a second live registration as well. Run this as a metastore admin:

SELECT table_catalog, table_schema, table_name, storage_path
FROM system.information_schema.tables
WHERE table_type = 'MANAGED'
  AND storage_path LIKE 'abfss://container@mystorageaccount.dfs.core.windows.net%';

Soft-deleted tables never appear there, and results are filtered to objects you have privileges on, so read a clean result narrowly.

Worth avoiding next time: Unity Catalog best practices advises against using "a single storage account and container for managed storage and other external locations", which is the shape that produced this.

[emorgoch]

Thanks Abhilash,

Unfortunately it is an external system that we can't control that's writing the files to the container root and that cannot be changed. As I wrote in my description, this container is not meant to have UC managed resources in it and it was done so accidentally on the external location setup. I'm manually copying files to a subdirectory as a temporary setup so I can continue development and testing, but will require that the root path to be made available for ingest. So I'm trying to figure out how to remove the block.

I did validate that there weren't any managed tables currently registered under the path using the query provided, and no results were returned.