topic Re: Error on Workflow: Failure to initialize configuration for storage account in Machine Learning

Error on Workflow: Failure to initialize configuration for storage account

Tingting — Sun, 22 Sep 2024 15:07:41 GMT

I have set up a workflow with a sequence of jobs. Each job run fine in an interactive mode, that is, run the notebook directly. However, when I tried to run the workflow, it got error on a step which uses a function from a Repo.

the error says "Failure to initialize configuration for storage account prod50a423bf8b426360bae3.dfs.core.windows.net: Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key"

I am certain that I have access to the Repo, the notebooks, the workflow.

What did I miss in the workflow?

Re: Error on Workflow: Failure to initialize configuration for storage account

filipniziol — Sun, 22 Sep 2024 16:12:46 GMT

Hi @Tingting ,

It seems that when you run notebook interactively, your personal credentials are used to access ADLS.

When the workflow job is run, Databricks uses different context.

Could you share whether your job is accessing some storage account, and how what type of authentication is used to access azure storage account? Are you the owner of the job? Are you using unity catalog?

Re: Error on Workflow: Failure to initialize configuration for storage account

Tingting — Sun, 22 Sep 2024 20:38:32 GMT

Thanks for the tip. The Repo does read data from delta tables, but not using unity catalog.
One try worked now. that is to switch the job cluster (although it's created by me) to an all-purpose one, and the workflow run fine.
I think it's about the access to Repo on ADO. I tried both using personal access token and Azure Azure Active Directory. the steps prior to use the Repo methods worked no mater job or all-purpose cluster used. If using job cluster, it always failed at the step.