topic dbutils.fs.ls doesn't work on Unity catalog managed location in Data Governance

dbutils.fs.ls doesn't work on Unity catalog managed location

venkad — Wed, 01 Mar 2023 09:11:29 GMT

Hi,

dbutils.fs.ls() is a most useful utility to list the files in remote cloud storage. I cannot use this function on top of the location managed by the Unity catalog.

I have a managed external location created on `'abfss://xyz@abcd.dfs.core.windows.net/` where my schema is created. When i try to list the files created by the tables inside this schema, i get this error.

```

%py display(dbutils.fs.ls('abfss://xyz@abcd.dfs.core.windows.net/__unitystorage/schemas/5551e5b7-dd3f-45cb-94ce-a572e3ff7760/tables/0fbd721d-1701-4516-ad09-d04a3057e016'))

```

Error message

```

AnalysisException: [RequestId=80fd99cc-efd3-44c3-86ae-17bbd5e699bb ErrorClass=INVALID_PARAMETER_VALUE.LOCATION_OVERLAP] Input path url 'abfss://xyz@abcd.dfs.core.windows.net/__unitystorage/schemas/5551e5b7-dd3f-45cb-94ce-a572e3ff7760/tables/0fbd721d-1701-4516-ad09-d04a3057e016' overlaps with managed storage

```

I tried to run **list** command as well. It works only on the exact location. But throws error when tries to list the directory inside it.

Below code works.

```

list 'abfss://xyz@abcd.dfs.core.windows.net/'

```

Below code doesn't work.

```

list 'abfss://xyz@abcd.dfs.core.windows.net/__unitystorage/'.

```

Is this an expected behavior? If yes, we are missing a great utility in the unity catalog.

Is there any work around or am I missing something?

Re: dbutils.fs.ls doesn't work on Unity catalog managed location

Anonymous — Sat, 11 Mar 2023 23:54:28 GMT

This is an expected behaviour. You trying to query the managed location of a catalog and this path is to be accessed by UC only, so abfss://xyz@abcd.dfs.core.windows.net/__unitystorage/* won't be accessible. Any subdirectories within xyz container will work.

list 'abfss://xyz@abcd.dfs.core.windows.net/<sub_directory> should work

Re: dbutils.fs.ls doesn't work on Unity catalog managed location

Anonymous — Thu, 30 Mar 2023 09:02:03 GMT

Hi @Venkadeshwaran K

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!