Frequent “GetPathStatus” and “GetBlobProperties” PathNotFound Errors on Azure Storage in Databricks

[h_h_ak]

We are encountering frequent GetPathStatus and GetBlobProperties errors when trying to access Azure Data Lake Storage (ADLS) paths through our Databricks environment. The errors consistently return a 404 PathNotFound status for paths that should be accessible.

Context:

• Operation: df.write() and df.read() operations on Databricks, attempting to access Azure storage paths.
• Storage Path: /stxxxx/src-sapecc/ and other related paths in Azure Data Lake Gen2.
• Errors Observed:
• GetPathStatus: PathNotFound
• GetBlobProperties: BlobNotFound

Error Count: The errors are recurring frequently, as seen in the attached logs, which indicate multiple instances of the PathNotFound error with status code 404.

Timestamps: Errors occur across multiple timestamps (see attached logs for details).

Attached Screenshot: Logs showing details of the error, including the operation name, status codes, and paths.

Could you please assist in identifying why these PathNotFound and BlobNotFound errors are occurring despite correct configuration and permissions? Additionally, if there’s any further configuration required on the Azure or Databricks side to resolve this, please advise.Thanks in advance..

[saurabh18cs]

Hi,

1) Ensure that the paths you are trying to access are correct and exist in the ADLS Gen2 storage account.

2)  Verify that the Databricks cluster has the necessary permissions to access the ADLS Gen2 paths

Br