Error while describing table created in unity catalog

[Shubhanshu]

I have created an external table in unity catalog and when describing its data getting the following error

org.apache.hadoop.fs.FileAlreadyExistsException: Operation failed: "The specified path, or an element of the path, exists and its resource type is invalid for this operation.", 409, GET, https://unitymetastorepoc.dfs.core.windows.net/unity-metastore?upn=false&resource=filesystem&maxResu..., PathConflict, "The specified path, or an element of the path, exists and its resource type is invalid for this operation. RequestId:e1187b37-301f-006d-0ec4-d18084000000 Time:2023-08-18T11:12:39.0536283Z"

What does this error mean and why does it occur?

[Kaniz · Accepted solution]

Hi @Shubhanshu , The error message you're seeing is a FileAlreadyExistsException from the Hadoop File System (HDFS). This error typically occurs when there's an attempt to create a file or directory that already exists, and the operation being performed is not compatible with the existing resource type. In your case, it's likely that the path you're trying to use for your external table in Unity Catalog already exists and is not suitable for the operation you're trying to perform.

This could be due to one of the following reasons:

1. **Incomplete Cloud Storage URI**: As per the given information, if you are creating an external Unity Catalog (UC) table and the complete Cloud Storage URI is not provided, you might encounter this error.

Ensure that the complete storage URI with scheme s3/abfss is provided when creating an external table. For example,
   

sql
    create test catalog1.schema1.emp1 location 's3://loc' as select * from employee
    

2. **Overlap with Managed Storage**: According to the given information, this error can also occur if you're trying to create an external table in a managed Unity Catalog location. Creating an external table in a managed path is not supported by design. You should use a non-overlapping path. 

Please review your operation and ensure you're using a complete and non-overlapping Cloud Storage URI path for your external table.