Frequent “GetPathStatus” and “GetBlobProperties” PathNotFound Errors on Azure Storage in Databricks

[h_h_ak]

We are encountering frequent GetPathStatus and GetBlobProperties errors when trying to access Azure Data Lake Storage (ADLS) paths through our Databricks environment. The errors consistently return a 404 PathNotFound status for paths that should be accessible.

Context:

• Operation: df.write() and df.read() operations on Databricks, attempting to access Azure storage paths.
• Storage Path: /stxxxx/src-sapecc/ and other related paths in Azure Data Lake Gen2.
• Errors Observed:
• GetPathStatus: PathNotFound
• GetBlobProperties: BlobNotFound

Error Count: The errors are recurring frequently, as seen in the attached logs, which indicate multiple instances of the PathNotFound error with status code 404.

Timestamps: Errors occur across multiple timestamps (see attached logs for details).

Attached Screenshot: Logs showing details of the error, including the operation name, status codes, and paths.

Could you please assist in identifying why these PathNotFound and BlobNotFound errors are occurring despite correct configuration and permissions? Additionally, if there’s any further configuration required on the Azure or Databricks side to resolve this, please advise.Thanks in advance..

[saurabh18cs]

Hi,

1) Ensure that the paths you are trying to access are correct and exist in the ADLS Gen2 storage account.

2)  Verify that the Databricks cluster has the necessary permissions to access the ADLS Gen2 paths

Br

[h_h_ak]

Hi,

I confirm that I have checked it, and everything seems to be in order—both path and permissions are definitely in place, as we are also successfully writing data to the container. I noticed that these messages come up in several situations:

1.In SQL Warehouse queries

2. During spark.read... operations

3. During spark.write... operations

We are using DBR 13.3 in this workspace. Any ideas on why so many storage-related messages are appearing? It only started happening after we enabled diagnostic settings in Azure.

[aashish122]

Testing this in incognito mode will help !!!

[h_h_ak]

Why do you think this will help? We have Spark and connection configuration in the cluster settings, and spark.read... or write statements are executed by the notebook. Additionally, the SQL queries are coming from outside. How would Incognito help in this scenario?