topic Re: Upload to Volume inside unity catalog not possible? in Data Engineering

Upload to Volume inside unity catalog not possible?

boriste — Fri, 28 Jul 2023 13:44:44 GMT

I want to upload a simple csv file to a volume which was created in our unity catalog. We are using secure cluster connectivity and our storage account (metastore) is not publicly accessable. We injected the storage in our vnet.

I am getting the following error while uploading:

"An error occurred while uploading. Please try again"
Should i open the IP Address of the control plane in our storage account? Is there a way to fix this?

Re: Upload to Volume inside unity catalog not possible?

karthik_p — Fri, 28 Jul 2023 15:49:43 GMT

@boriste can you please check if you have permissions to read/write to volume, or you admin if not please reach out to admin to provide to access to write/read to your volume

Re: Upload to Volume inside unity catalog not possible?

boriste — Mon, 31 Jul 2023 06:11:31 GMT

@karthik_p I have full admin rights and im logged in with my azure ad account. I also have read/write permissions

But im getting the message:

{
  "error_code" : "BAD_REQUEST",
  "message" : "Missing credentials to access the DBFS root storage container in Azure."
}

I dont know if databricks uses another identity but somehow i cant upload

Re: Upload to Volume inside unity catalog not possible?

Martinitus — Mon, 31 Jul 2023 08:43:00 GMT

We (boriste and me) have tested creating volumes from a notebook which worked just fine. This means, networking and auth from the cluster (dataplane) to the metastore storage account works.

What was not working was uploading a file via the Web-UI. Our suspicion is, that this upload tries to save the file to the metastore storage account from the databricks control plane. Which does _not_ have access (networking is blocked by NSG) to our metastore storage account.

I assume one solution would be to whitelist the IP of the databricks controlplane in the networking settings of our metastore storage account. We could try that but we don't know the appropriate IP range.

Re: Upload to Volume inside unity catalog not possible?

Martinitus — Mon, 31 Jul 2023 09:15:28 GMT

We checked the storage account logs of the metastore. Whenever we try to upload a file to a volume from the UI of the workspace, we get an authorization error. The request seems to be coming from a Databricks Java Backend service (ControlPlane?). We tried adding the caller IP to the storage account networking white list - that is not possible because its not a public IP?

Re: Upload to Volume inside unity catalog not possible?

AdrianaIspas — Tue, 01 Aug 2023 13:29:53 GMT

Hi @Martinitus

I believe this might be related to one of the known limitations listed in our documentation:
"You cannot upload or download files for volumes backed by Azure storage accounts configured with Azure Firewall or Private Link" Source

We are actively working on this and hope for a fix in the coming weeks.

Re: Upload to Volume inside unity catalog not possible?

jeroenvs — Mon, 09 Oct 2023 11:38:07 GMT

@AdrianaIspas We are running into the same issue. It took a while to figure out that the error message is related to this limitation. Any updates on when we can expect the limitation to be taken away? We want to secure access to our storage accounts with a firewall, setting up a public storage account is not an option.

Re: Upload to Volume inside unity catalog not possible?

UstOldfield — Thu, 12 Oct 2023 10:57:14 GMT

Hello! Do you have an idea as to when this will be fixed - what release is it aligned to? As this is still an issue and is causing me and my customers some issues, considering that you can't use DBFS with Unity Catalog.

Re: Upload to Volume inside unity catalog not possible?

AdrianaIspas — Tue, 24 Oct 2023 14:13:06 GMT

Hi @UstOldfield

A fix was released for the previous issues where upload/download was not working for Azure storage accounts configured with Azure Firewall or Private Link.

Can you try to see if this addresses your problem?

Re: Upload to Volume inside unity catalog not possible?

UstOldfield — Tue, 24 Oct 2023 14:32:30 GMT

Hey @AdrianaIspas - it works. Thanks for letting me know.

Re: Upload to Volume inside unity catalog not possible?

AdrianaIspas — Tue, 24 Oct 2023 22:20:01 GMT

Hi @jeroenvs

A fix was released for the previous issue where upload/download was not working for Azure storage accounts configured with Azure Firewall or Private Link.

Can you try to see if this addresses your problem?

Re: Upload to Volume inside unity catalog not possible?

AdrianaIspas — Tue, 24 Oct 2023 22:21:38 GMT

A fix was released for the previous issues where upload/download was not working for Azure storage accounts configured with Azure Firewall or Private Link.