Using Autoloader in DLT: ErrorClass=INVALID_PARAMETER_VALUE.LOCATION_OVERLAP]

lprevost — Mon, 14 Oct 2024 21:07:57 GMT

I've been using Autloloader in a DLT pipeline loading data from an s3 location to my hive_metastore shared with AWS glue.

I'm now trying to migrate this over to Unity Catalog to take advantage of liquid clustering and data quality.

However, I'm getting the following error when I run my DLT pipeline:

AnalysisException: [RequestId=54d8b129-2dfb-46cc-9c42-4cab3ed6349d ErrorClass=INVALID_PARAMETER_VALUE.LOCATION_OVERLAP] Input path url 's3://sled-lake/sl_unity/__unitystorage/schemas/f07b0eee-2481-4803-b31d-f45fa37f03f9/tables/bd6ecf6f-f4af-47ac-bfff-d8dc87418522/_dlt_metadata/_autoloader' overlaps with managed storage within 'GenerateTemporaryPathCredential' call. .,None,Map(),Map(),List(),List(),Map())

I do understand that external locations must be mapped to a point lower in the directory structure of where my unity catalog is stored.

My unity catalog is setup to a location on my s3 as s3://sled-lake/sl_unity

Below that, I have catalogs prod and dev. And within prod, I have created a Volume mapped to my external data which is on s3://commoncrawl.

I do think there may have been some external locations that conflicted but they have been deleted. Could it be I'm running into something in garbage collection for deletion?

Re: Using Autoloader in DLT: ErrorClass=INVALID_PARAMETER_VALUE.LOCATION_OVERLAP]

NandiniN — Fri, 01 Nov 2024 06:10:34 GMT

https://kb.databricks.com/unity-catalog/invalid_parameter_valuelocation_overlap-overlaps-with-managed-storage-error

topic Using Autoloader in DLT: ErrorClass=INVALID_PARAMETER_VALUE.LOCATION_OVERLAP] in Data Engineering

Using Autoloader in DLT: ErrorClass=INVALID_PARAMETER_VALUE.LOCATION_OVERLAP]

Re: Using Autoloader in DLT: ErrorClass=INVALID_PARAMETER_VALUE.LOCATION_OVERLAP]