topic Re: Connect to Blob storage "no credentials found for them in the configuration" in Data Engineering

Connect to Blob storage "no credentials found for them in the configuration"

winston12 — Thu, 13 Sep 2018 16:56:40 GMT

I'm working with Databricks notebook backed by spark cluster. Having trouble trying to connect to the Azure blob storage. I used this link and tried the section Access Azure Blob Storage Directly - Set up an account access key. I get no errors here:

spark.conf.set(  "fs.azure.account.key.<your-storage-account-name>.blob.core.windows.net",  "<your-storage-account-access-key>")

But receive errors when I try and do an 'ls' on the directory:

dbutils.fs.ls("wasbs://<your-container-name>@<your-storage-account-name>.blob.core.windows.net/<your-directory-name>")

shaded.databricks.org.apache.hadoop.fs.azure.AzureException: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Unable to access container <container name> in account <storage account name>core.windows.net using anonymous credentials, and no credentials found for them in the configuration.

Re: Connect to Blob storage "no credentials found for them in the configuration"

statueofmike — Mon, 12 Nov 2018 18:10:08 GMT

I get the same error when trying to follow this DataBricks example. It doesn't seem to be using the access credentials correctly, but I don't know how to fix this problem.

Re: Connect to Blob storage "no credentials found for them in the configuration"

adipolak — Tue, 19 Mar 2019 16:15:06 GMT

Happened to me as well and I discovered that the storage and the databricks cluster were in different region, so I moved the storage. This fixed the issue and now it works!

Re: Connect to Blob storage "no credentials found for them in the configuration"

DonaldLeckie — Fri, 26 Jul 2019 10:43:27 GMT

I had the same problem, but was connecting via RDD API (I was using my own java adaptor to azure blobs). The https://docs.databricks.com/spark/latest/data-sources/azure/azure-storage.html page states at the bottom that you should create your cluster with hadoop config containing the credentials - either access key or SAS token.

I tried with the access key (I'm just experimenting) and that started to work just fine.

@annashetty you might want to try the same.

Re: Connect to Blob storage "no credentials found for them in the configuration"

MarkMcLaughlin — Thu, 04 Feb 2021 05:09:09 GMT

I had this problem bc I copy-pasted the header key and did not replace the one <storage-account-name> to the other. That's my fault but... it is a very long header key name if your storage account name is kinda generic.

fs.azure.account.key.<STORAGE-ACCOUNT-NAME>.blob.core.windows.net

Re: Connect to Blob storage "no credentials found for them in the configuration"

Feder — Tue, 27 Apr 2021 14:25:36 GMT

I have been facing the same problem over and over. Now trying to follow what's written here (https://docs.databricks.com/data/data-sources/azure/azure-storage.html#access-azure-blob-storage-directly), but always getting "shaded.databricks.org.apache.hadoop.fs.azure.AzureException: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Container containerfede in account storageaccountfede.blob.core.windows.net not found, and we can't create it using anoynomous credentials, and no credentials found for them in the configuration". Both my cluster and blob storage are located in the same place, so I am assuming that it must be the cluster's configuration.