topic Re: External location + Failure to initialize configuration for storage account in Data Engineering

External location + Failure to initialize configuration for storage account

RYBK — Fri, 29 Sep 2023 09:20:08 GMT

Hello,
I created a storage credential and an external location. Test is ok, I'm able to browse it from the portal.
I have a notebook to create a table :
%sql
CREATE OR REPLACE TABLE myschema.mytable
(
data1 string,
data2 string
)
USING DELTA LOCATION "abfss://mycontainer@myaccount.blob.core.windows.net/";

When I'm executing the notebook with a SQL warehouses cluster, it works fine ; with a standard cluster it failed with this error "Failure to initialize configuration for storage account myaccount.dfs.core.windows.net: Invalid configuration value detected for fs.azure.account.key"

I thought it was no longer needed to set fs.azure.account.key. with an external location, am I wrong ? Am I missing something ?

Re: External location + Failure to initialize configuration for storage account

RYBK — Thu, 05 Oct 2023 11:32:24 GMT

Hello @Retired_mod

Thank you for your response. We found the issue. We use a ML Cluster with "no isolation shared" as access mode. This cluster is not compatible with unity. We set fs.azure.* variables in the cluster conf. like below to fix our issue:

fs.azure.account.oauth2.client.id.mydatalake.dfs.core.windows.net: {{secrets/mysecret/myclientid}}.

Is there a better/proper way to do it ?

Regards

Re: External location + Failure to initialize configuration for storage account

RYBK — Thu, 05 Oct 2023 11:45:17 GMT

I believe this is what we are doing, as I mentioned in my example, right?

Thank you