External location + Failure to initialize configuration for storage account

[RYBK]

Hello,
I created a storage credential and an external location. Test is ok, I'm able to browse it from the portal. 
I have a notebook to create a table :
%sql
CREATE OR REPLACE TABLE myschema.mytable
(
  data1 string,
 data2 string
)
USING DELTA LOCATION "abfss://mycontainer@myaccount.blob.core.windows.net/";
 
When I'm executing the notebook with a SQL warehouses cluster, it works fine ; with a standard cluster it failed with this error "Failure to initialize configuration for storage account myaccount.dfs.core.windows.net: Invalid configuration value detected for fs.azure.account.key"
 
I thought it was no longer needed to set fs.azure.account.key. with an external location, am I wrong ? Am I missing something ?
 

[Kaniz]

Hi @RYBK, Based on your provided information, you're trying to create a table in Databricks using an Azure Blob Storage (ABFS) location. Your error message suggests an issue with your storage account's configuration. Although you have created an external site, and it is testable from the portal, it seems like the Databricks cluster cannot access the data because it fs.azure.account.key is not set or is invalid. While it is true that you can use an external location without specifying fs.azure.account.key in some cases, it seems like your standard cluster still requires it. 

Here's what you can do to resolve the issue.

1. Ensure that the storage account key (fs.azure.account.key) is correctly set in your Databricks workspace. You can set this in the DBFS configurations.
2. Verify that the storage account key is correct. You can find this key in the Azure portal under the settings of your storage account.
3. Check if the storage account and the Databricks workspace are in the same region. If not, you may need to create a VNet service endpoint.

The exact steps can vary depending on your specific setup and configuration.

If the issue persists, I recommend contacting Databricks support for further assistance.

[RYBK · Accepted solution]

Hello @Kaniz 

Thank you for your response. We found the issue. We use a ML Cluster with "no isolation shared" as access mode. This cluster is not compatible with unity. We set fs.azure.* variables in the cluster conf. like below to fix our issue:

fs.azure.account.oauth2.client.id.mydatalake.dfs.core.windows.net: {{secrets/mysecret/myclientid}}. 

Is there a better/proper way to do it ?

Regards

[Kaniz]

Hi @RYBK, The method you're using to set the fs.azure.* variables in the cluster configuration is a common way to handle Azure data lake Storage Gen2 configurations in Databricks. 

However, if you're looking for a more secure and centralized way to manage these configurations, you might consider using Databricks secrets. Secrets in Databricks are a safe way to store and use sensitive information like access keys, passwords, or connection strings. 

You can create a secret scope and store your fs.azure.* values as secrets. Then, you can reference these secrets directly in your cluster configuration. This way, the sensitive information is not exposed in the cluster configuration, and you can manage all your secrets in one place.

[RYBK]

I believe this is what we are doing, as I mentioned in my example, right?

Thank you

[Kaniz]

Yes, that's the best way to do it.