cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Error on Workflow: Failure to initialize configuration for storage account

Tingting
New Contributor III

I have set up a workflow with a sequence of jobs. Each job run fine in an interactive mode, that is, run the notebook directly. However, when I tried to run the workflow, it got error on a step which uses a function from a Repo. 

the error says "Failure to initialize configuration for storage account prod50a423bf8b426360bae3.dfs.core.windows.net: Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key"

I am certain that I have access to the Repo, the notebooks, the workflow.

What did I miss in the workflow?

2 REPLIES 2

filipniziol
Contributor III

Hi @Tingting ,

It seems that when you run notebook interactively, your personal credentials are used to access ADLS.

When the workflow job is run, Databricks uses different context. 

Could you share whether your job is accessing some storage account, and how what type of authentication is used to access azure storage account? Are you the owner of the job? Are you using unity catalog?

 

 

Tingting
New Contributor III

Thanks for the tip. The Repo does read data from delta tables, but not using unity catalog. 
One try worked now. that is to switch the job cluster (although it's created by me) to an all-purpose one, and the workflow run fine. 
I think it's about the access to Repo on ADO. I tried both using personal access token and Azure Azure Active Directory. the steps prior to use the Repo methods worked no mater job or all-purpose cluster used. If using job cluster, it always failed at the step.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group