Dear Databricks Community,
I am reaching out to you for assistance with some issues I'm encountering in my Databricks environment. I'm hoping the community can provide some guidance to help me resolve these problems.
1. Error with df.display(): When I try to use the df.display() function, I'm receiving the following error:
"'Failed to store the result. Try rerunning the command. Failed to upload command result to DBFS. Error message: PUT request to create file error HttpResponse Proxy(HTTP/1.1 404 The specified filesystem does not exist. [Content-Length: 175. Content-Type: application/json;charset=utf-8, Server: Windows-Azure-HDFS/1.0 Microsoft-HTTPAPI/2.0, x-ms-error-code: FilesystemNotFound, x-ms-request-id: 33854442-8011- 0028-3da6-bc0285000000, x-ms-version: 2021-04-10, Date: Wed, 12 Jun 2024 08:59:45 GMT] Response Entity Proxy([Content-Type: application/json;charset=utf-8 Content-Length: 175, Chunked: false])) "
2. Error with df.show():
I'm also facing a similar issue when using df.show() for data frames with more than 10-20 rows. The same error as mentioned in point 1 appears.
Debugging insights:
1. During my investigation, I found that the issue seems to be related to Databricks trying to access the root storage and failing to write the data.
error on cluster logs: '' ERROR PresignedUrlClientUtils$: FS_OP_CREATE FILE[https://<root_storage-account-name>.dfs.core.windows.net/jobs/4079550013704479
/command-results/4056370604825597/b3041abc-2eeb-45da-90b7-23d98973d4d0] Presigned URL: Failed to upload stream using AzureAdl2SasUri"
2. I tried to upload some files to DBFS (Databricks File System) from the UI, but it's throwing an error that says 'The Azure Container Does Not exist' and also a 500 error code.
3. Same error faced when I tried to import 3 to 4 notebook on shared location
4. I tried with different different types of clusters and databricks run time also but same issues
5. I have Admin access on databricks workspace and I am using Premium (+ Role-based access controls) databricks
6. Currently I am using another storage account for my DE Work using storage credentials I am able to access data easily in Notebooks
I'm not sure how to resolve this problem, as the root storage is inside the managed resource group of Databricks.
I would greatly appreciate if the community could provide any insights, suggestions, or solutions to help me address these issues. Your assistance would be invaluable in getting my Databricks environment back on track.
Thank you in advance for your time and support.
Best regards,
Vishal Yadav