External locations

[Snoonan]

Hi all,

I am getting an error when I try to 'Browse' an external location in catalog explorer through Databricks UI.

This is the error I get:

Error loading files.

Input path url 'abfss://dbx-ccc-hive-metastore-unity-catalog@xxxxxxxxxx.dfs.core.windows.net/__unitystorage' overlaps with managed storage within 'ListFiles' call

Has anyone seen this error before and/or understands what is going wrong?

Thanks,

Sean

[NandiniN · Accepted solution]

Hi Sean,

This is expected for a managed path, as a user, we are not supposed to directly access paths that are managed by UC.

Hope you are able to access this KB with detailed info - https://kb.databricks.com/unity-catalog/invalid_parameter_valuelocation_overlap-overlaps-with-manage...

Thanks!

[EmilianoLlera_D]

Hi NandiniN,

I'm having the same error. Let me give you some context of my use case.

I need to calculate the size of the container, so I created a function that loops over the folder structure using "dbutils.fs.ls". Inside container we have both, external and managed tables. But I'm not attempting to create anything inside container, just need to read every file size inside container. When attempting to do so, is that I'm getting below error.

===== ERROR =====
[RequestId=887e131d-4cea-41f6-a226-7b804e07f938 ErrorClass=INVALID_PARAMETER_VALUE.LOCATION_OVERLAP] Input path url 'abfss://<container>@<storage_account>.dfs.core.windows.net/managed/<subfolder>/data/__unitystorage' overlaps with managed storage within 'ListFiles' call. 

I've assigned permissions "Storage Blob Data Contributor" + "Storage Blob Data Owner" on storage account to the  Managed Identity declared in Catalog. I've also added permissions "ALL PRIVILEGES" on External location and Storage Credentials. But I'm still getting the same error.

Is this supported in UC? If this is not the best approach, how would you achieve this task?

Regards, Emiliano

[Snoonan · Accepted solution]

Hi @NandiniN ,

Thank you for quashing my concerns.

Thanks,

Sean

[gmiguel]

Adding some tips...
It seems that you have created a metastore with root path definition (metastore-level managed storage). If you're planning to deploy a multi-workspace environment (Development, QA and Production), I strongly recommend you to change this configuration to Catalog-level storage. 

Since in general those kind of environments should be isolated, you will have tough days as soon as it gets bigger when using metastore-level storage.

[Snoonan]

Hi @gmiguel ,

Thank you for the advice.

Is this a setting when creating the metastore, or do I simply not specify an external location for metastore and only do so for catalogs?

Thanks,

Sean

[gmiguel]

Simply do not specify a location during metastore creation. Do it at catalog level and schema level (optional).

[lprevost]

I'm running into this error when only using Autoloader within my UC.   So, my assumption is that when Autoloader does  a file listing for my source files, it creates the error.   I've tried pointing Autloloader to an s3 location (a public bucket) and also a /Volume path that is mapped into my unity catalog.  Both generate the same error.

I have no problems with a similar attempt outside of UC using my hive_metastore which I share with my AWS instance.

My metastore path is unspecified but my catalog path is specified.

[NandiniN]

Verify that the necessary permissions are granted for the catalog location. This includes ensuring that the storage credentials are correctly configured and that you have the appropriate read/write permissions.

[NandiniN]

Also, as mentioned in the kb https://kb.databricks.com/unity-catalog/invalid_parameter_valuelocation_overlap-overlaps-with-manage...

You can create an external location under some-root/ as long as it does not overlap the managed table. Given the example, <storage-blob>://<some-root>/<some-path>/<external-table-path> is a valid path for an external location.

If you try to list the contents of this example location, the result would be successful.