cancel
Showing results for 
Search instead for 
Did you mean: 
Data Governance
Join discussions on data governance practices, compliance, and security within the Databricks Community. Exchange strategies and insights to ensure data integrity and regulatory compliance.
cancel
Showing results for 
Search instead for 
Did you mean: 

dbutils.fs.ls doesn't work on Unity catalog managed location

venkad
Contributor

Hi,

dbutils.fs.ls() is a most useful utility to list the files in remote cloud storage. I cannot use this function on top of the location managed by the Unity catalog.

I have a managed external location created on `'abfss://xyz@abcd.dfs.core.windows.net/` where my schema is created. When i try to list the files created by the tables inside this schema, i get this error.

```

%py display(dbutils.fs.ls('abfss://xyz@abcd.dfs.core.windows.net/__unitystorage/schemas/5551e5b7-dd3f-45cb-94ce-a572e3ff7760/tables/0fbd721d-1701-4516-ad09-d04a3057e016'))

```

Error message

```

AnalysisException: [RequestId=80fd99cc-efd3-44c3-86ae-17bbd5e699bb ErrorClass=INVALID_PARAMETER_VALUE.LOCATION_OVERLAP] Input path url 'abfss://xyz@abcd.dfs.core.windows.net/__unitystorage/schemas/5551e5b7-dd3f-45cb-94ce-a572e3ff7760/tables/0fbd721d-1701-4516-ad09-d04a3057e016' overlaps with managed storage

```

I tried to run **list** command as well. It works only on the exact location. But throws error when tries to list the directory inside it.

Below code works.

```

list 'abfss://xyz@abcd.dfs.core.windows.net/'

```

Below code doesn't work.

```

list 'abfss://xyz@abcd.dfs.core.windows.net/__unitystorage/'.

```

Is this an expected behavior? If yes, we are missing a great utility in the unity catalog.

Is there any work around or am I missing something?

3 REPLIES 3

Kaniz_Fatma
Community Manager
Community Manager

Hi @Venkadeshwaran K​, The error message you received is an analysis exception that indicates an issue with the input path URL you provided. Specifically, the error message suggests that the input path URL overlaps with managed storage.

To resolve this issue, you may need to adjust the path URL you are using to access the data. You could try changing the course URL. Additionally, you may want to check the permissions and access controls on the data you are attempting to access to ensure that you have the necessary privileges to access it.

Anonymous
Not applicable

This is an expected behaviour. You trying to query the managed location of a catalog and this path is to be accessed by UC only, so abfss://xyz@abcd.dfs.core.windows.net/__unitystorage/* won't be accessible. Any subdirectories within xyz container will work.

list 'abfss://xyz@abcd.dfs.core.windows.net/<sub_directory> should work

Anonymous
Not applicable

Hi @Venkadeshwaran K​ 

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance! 

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!