topic Re: Databricks Azure Blob Storage access in Data Engineering

Databricks Azure Blob Storage access

tariq — Wed, 26 Oct 2022 05:43:55 GMT

I am trying to access files stored in Azure blob storage and have followed the documentation linked below:

https://docs.databricks.com/external-data/azure-storage.html

I was successful in mounting the Azure blob storage on dbfs but it seems that the method is not recommended anymore. So, I tried to set up direct access using URI after using SAS authentication.

spark.conf.set("fs.azure.account.auth.type.<storage-account>.dfs.core.windows.net", "SAS")
spark.conf.set("fs.azure.sas.token.provider.type.<storage-account>.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider")
spark.conf.set("fs.azure.sas.fixed.token.<storage-account>.dfs.core.windows.net", "<token>")

Now when I try to access any file using:

spark.read.load("abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/<path-to-data>")

I get the following error:

Operation failed: "Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.", 403, HEAD,

What needs to be changed for this to work?

Re: Databricks Azure Blob Storage access

Anonymous — Mon, 28 Nov 2022 04:24:59 GMT

Hi @Tarique Anwar

Does @Debayan Mukherjee response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

We'd love to hear from you.

Thanks!

Re: Databricks Azure Blob Storage access

ravinchi — Fri, 13 Jan 2023 07:17:57 GMT

@Debayan Mukherjee @Tarique Anwar @Vidula Khanna

Even we're facing this issue while creating table from databricks sql.

create table test using delta location 'abfss://[container_name]@[storage_account].dfs.core.windows.net/'

We created external_location, storage_credentail with access_connector id. And provided storage blob data contributor role for access conenctor id. It was working until day before yesterday, not from yesterday. Getting below error

"AuthenticationFailed, "Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature"

I followed the link shared- https://learn.microsoft.com/en-us/answers/questions/334786/azure-blob-storage-fails-to-authenticate-34make-su.html along with the stack over flow - https://stackoverflow.com/questions/24492790/azurestorage-blob-server-failed-to-authenticate-the-request-make-sure-the-value which is pointing but still no luck.

if we have to control api version in header, isn't it something in control of databricks? as we're using databricks sql.

Re: Databricks Azure Blob Storage access

ravinchi — Tue, 17 Jan 2023 09:21:53 GMT

Firewall settings are fine, and its allowed from all networks @Debayan Mukherjee

Re: Databricks Azure Blob Storage access

Debayan — Wed, 26 Oct 2022 07:10:32 GMT

Hi @Tarique Anwar , could you please refer to https://learn.microsoft.com/en-us/answers/questions/334786/azure-blob-storage-fails-to-authenticate-34make-su.html

Please let us know if this helps?

Re: Databricks Azure Blob Storage access

Debayan — Mon, 16 Jan 2023 04:35:08 GMT

Hi, @Ravindra Ch , could you please check the firewall settings in Azure networking?