topic Re: DBFS_DOWN in Data Engineering

DBFS_DOWN

markbaas — Thu, 16 Jan 2025 16:15:06 GMT

I have an Azure Databricks workspace with Unity Catalog setup, using VNet and private endpoints. Serverless works great; however, the regular clusters have problems showing large results:

Failed to store the result. Try rerunning the command.

Failed to upload command result to DBFS. Error message: PUT request to create file failed with statusCode=403, error=HttpResponseProxy{HTTP/1.1 403 This request is not authorized to perform this operation.

Also, I can’t list (ls) into DBFS. The event log shows:

DBFS_DOWN.

Perhaps this is an actual Databricks issue. My network and firewall setup has been validated many times. Somehow, the cluster has no access to the DBFS root. But as this is a Databricks-managed resource group, it should all work out of the box, right?

Re: DBFS_DOWN

Walter_C — Thu, 16 Jan 2025 16:31:03 GMT

What access mode is being used on this clusters?

Re: DBFS_DOWN

markbaas — Thu, 16 Jan 2025 16:31:57 GMT

Unrestricted and Single User mode.

Re: DBFS_DOWN

Walter_C — Thu, 16 Jan 2025 16:38:27 GMT

Can you please try the same in a shared access mode cluster?

Also can you please try setting up spark configuration spark.databricks.driver.enableWriteDbfsCommandResultInDp to false.

This will disable the feature that writes DBFS command results directly in the Data Plane

Re: DBFS_DOWN

markbaas — Thu, 16 Jan 2025 18:07:17 GMT

Function display and show work now again thanks!

The dbutils.fs.ls("dbfs:/") command still results in an error. I really wonder how databricks has setup those managed resources. I think something is going wrong there.

Re: DBFS_DOWN

markbaas — Fri, 17 Jan 2025 10:43:56 GMT

From spark and dbutils I also get this message:

Caused by: com.microsoft.azure.storage.StorageException: This request is not authorized to perform this operation.

Re: DBFS_DOWN

markbaas — Fri, 17 Jan 2025 10:44:52 GMT

I don't understand how the cluster authenticates with the storage account, perhaps if someone at databricks could clear this up for me I would be able to better debug the issue.

Re: DBFS_DOWN

markbaas — Mon, 20 Jan 2025 14:02:08 GMT

I solved the issue myself. Databricks documentation is hard, but it seems necessary to create private endpoints in the managed storage account of databricks.

Re: DBFS_DOWN

LeandroDias — Thu, 30 Jan 2025 11:39:44 GMT

I'm having the same issue when i try to save a large delta table (80M of rows). Could you please share how do you solved the problem?

Re: DBFS_DOWN

markbaas — Fri, 31 Jan 2025 09:33:38 GMT

The dbfs (dbstorage) resource in the managed azure resource group needs to have private endpoints to your virtual network. You can create those manually or through iac (bicep/terraform).