topic Re: External locations in Data Governance

External locations

Snoonan — Fri, 26 Apr 2024 07:35:13 GMT

Hi all,

I am getting an error when I try to 'Browse' an external location in catalog explorer through Databricks UI.

This is the error I get:

Error loading files.

Input path url 'abfss://dbx-ccc-hive-metastore-unity-catalog@xxxxxxxxxx.dfs.core.windows.net/__unitystorage' overlaps with managed storage within 'ListFiles' call

Has anyone seen this error before and/or understands what is going wrong?

Thanks,

Sean

Re: External locations

NandiniN — Fri, 26 Apr 2024 07:42:42 GMT

Hi Sean,

This is expected for a managed path, as a user, we are not supposed to directly access paths that are managed by UC.

Hope you are able to access this KB with detailed info - https://kb.databricks.com/unity-catalog/invalid_parameter_valuelocation_overlap-overlaps-with-managed-storage-error

Thanks!

Re: External locations

Snoonan — Mon, 29 Apr 2024 18:00:57 GMT

Hi @NandiniN ,

Thank you for quashing my concerns.

Thanks,

Sean

Re: External locations

gmiguel — Mon, 29 Apr 2024 19:08:52 GMT

Adding some tips...
It seems that you have created a metastore with root path definition (metastore-level managed storage). If you're planning to deploy a multi-workspace environment (Development, QA and Production), I strongly recommend you to change this configuration to Catalog-level storage.

Since in general those kind of environments should be isolated, you will have tough days as soon as it gets bigger when using metastore-level storage.

Re: External locations

Snoonan — Mon, 29 Apr 2024 21:08:01 GMT

Hi @gmiguel ,

Thank you for the advice.

Is this a setting when creating the metastore, or do I simply not specify an external location for metastore and only do so for catalogs?

Thanks,

Sean

Re: External locations

gmiguel — Mon, 29 Apr 2024 22:14:39 GMT

Simply do not specify a location during metastore creation. Do it at catalog level and schema level (optional).

Re: External locations

EmilianoLlera_D — Mon, 19 Aug 2024 14:25:43 GMT

Hi NandiniN,

I'm having the same error. Let me give you some context of my use case.

I need to calculate the size of the container, so I created a function that loops over the folder structure using "dbutils.fs.ls". Inside container we have both, external and managed tables. But I'm not attempting to create anything inside container, just need to read every file size inside container. When attempting to do so, is that I'm getting below error.

===== ERROR =====
[RequestId=887e131d-4cea-41f6-a226-7b804e07f938 ErrorClass=INVALID_PARAMETER_VALUE.LOCATION_OVERLAP] Input path url 'abfss://<container>@<storage_account>.dfs.core.windows.net/managed/<subfolder>/data/__unitystorage' overlaps with managed storage within 'ListFiles' call.

I've assigned permissions "Storage Blob Data Contributor" + "Storage Blob Data Owner" on storage account to the Managed Identity declared in Catalog. I've also added permissions "ALL PRIVILEGES" on External location and Storage Credentials. But I'm still getting the same error.

Is this supported in UC? If this is not the best approach, how would you achieve this task?

Regards, Emiliano

Re: External locations

lprevost — Tue, 15 Oct 2024 14:16:16 GMT

I'm running into this error when only using Autoloader within my UC. So, my assumption is that when Autoloader does a file listing for my source files, it creates the error. I've tried pointing Autloloader to an s3 location (a public bucket) and also a /Volume path that is mapped into my unity catalog. Both generate the same error.

I have no problems with a similar attempt outside of UC using my hive_metastore which I share with my AWS instance.

My metastore path is unspecified but my catalog path is specified.

Re: External locations

NandiniN — Thu, 31 Oct 2024 09:22:39 GMT

Verify that the necessary permissions are granted for the catalog location. This includes ensuring that the storage credentials are correctly configured and that you have the appropriate read/write permissions.

Re: External locations

NandiniN — Thu, 31 Oct 2024 09:24:10 GMT

Also, as mentioned in the kb https://kb.databricks.com/unity-catalog/invalid_parameter_valuelocation_overlap-overlaps-with-managed-storage-error

You can create an external location under some-root/ as long as it does not overlap the managed table. Given the example, <storage-blob>://<some-root>/<some-path>/<external-table-path> is a valid path for an external location.

If you try to list the contents of this example location, the result would be successful.