Using Autoloader in DLT: ErrorClass=INVALID_PARAMETER_VALUE.LOCATION_OVERLAP]

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.

I've been using Autloloader in a DLT pipeline loading data from an s3 location to my hive_metastore shared with AWS glue.

I'm now trying to migrate this over to Unity Catalog to take advantage of liquid clustering and data quality.

However, I'm getting the following error when I run my DLT pipeline:

AnalysisException: [RequestId=54d8b129-2dfb-46cc-9c42-4cab3ed6349d ErrorClass=INVALID_PARAMETER_VALUE.LOCATION_OVERLAP] Input path url 's3://sled-lake/sl_unity/__unitystorage/schemas/f07b0eee-2481-4803-b31d-f45fa37f03f9/tables/bd6ecf6f-f4af-47ac-bfff-d8dc87418522/_dlt_metadata/_autoloader' overlaps with managed storage within 'GenerateTemporaryPathCredential' call. .,None,Map(),Map(),List(),List(),Map())

I do understand that external locations must be mapped to a point lower in the directory structure of where my unity catalog is stored.

My unity catalog is setup to a location on my s3 as s3://sled-lake/sl_unity

Below that, I have catalogs prod and dev. And within prod, I have created a Volume mapped to my external data which is on s3://commoncrawl.

I do think there may have been some external locations that conflicted but they have been deleted. Could it be I'm running into something in garbage collection for deletion?