topic Re: strange error with dbutils.notebook.run(...) in Data Engineering

strange error with dbutils.notebook.run(...)

Mirko — Mon, 17 Jan 2022 11:41:31 GMT

The situation is as following: i have a sheduled job, which uses dbutils.notebook.run(path,timeout) . During the last week everything worked smooth. During the weekend the job began to fail, at the dbutils.notebook.run(path,timeout) command. I get the following error:

com.databricks.WorkflowException: com.databricks.NotebookExecutionException: FAILED: assertion failed: Attempted to set keys (credentials) in the extraContext, but these keys were not in the set of valid keys: {commandResultJsonMaxBytes, displayRowLimitV2, notebook_path, thresholdForStoringInDbfs, enableStoringResultsInDbfs, api_url, aclPathOfAclRoot, displayByteLimitV2}

to be honest, i am not 100% sure if my hoster changed anything in the databricks config during the weekend, but maybe somebody knows which situation would result in such an error.

Re: strange error with dbutils.notebook.run(...)

User16753724663 — Mon, 17 Jan 2022 12:37:15 GMT

Hi @Mirko Ludewig

This issue seems to be with High Concurrency cluster. Could you please confirm if you are using high concurrency cluster?.

Re: strange error with dbutils.notebook.run(...)

ShaikMaheer — Tue, 18 Jan 2022 06:46:09 GMT

Yes this issue faced in high concurrency Cluster when pass through is enabled in Azure data bricks. Any insights about this issue will be helpful?

Re: strange error with dbutils.notebook.run(...)

Mirko — Tue, 18 Jan 2022 07:54:36 GMT

@DARSHAN BARGAL @Shaik Maheer first, exsuce me for the delayed answer. I Asked my admin to create a high con cluster for me as well, because i wanted to check both cases. In fact i used a standard cluster all the time. Strangely today everything works fine. With both high con and standaard clusters. I guess that is a good thing

Re: strange error with dbutils.notebook.run(...)

User16753724663 — Tue, 18 Jan 2022 08:01:14 GMT

Hi @Mirko Ludewig @Shaik Maheer ,

Thank you for the update. We applied the fix from the backend to resolve this issue. We apologies for the inconvenience caused here. I hope everything is working fine now at your end.

Re: strange error with dbutils.notebook.run(...)

Mirko — Tue, 18 Jan 2022 08:23:57 GMT

@DARSHAN BARGAL thank you for the explanation. It is good to know that i can recieve help quickly here!

Re: strange error with dbutils.notebook.run(...)

florent — Tue, 18 Jan 2022 11:32:06 GMT

Hi @DARSHAN BARGAL ,

we have the same problem and it is still present at 11:31 utc

Cluster : standard

Passtrough :on

Runtime : 9.1 ml

region : azure francecentral

Re: strange error with dbutils.notebook.run(...)

Mirko — Tue, 18 Jan 2022 11:37:27 GMT

@Florent POUSSEROT my check was on runtime 10.1

Re: strange error with dbutils.notebook.run(...)

KC_Databricks — Wed, 19 Jan 2022 02:27:39 GMT

@DARSHAN BARGAL

I am facing a similar error at a client location today (19Jan2022). This is on a pass-thru high con cluster. What was the fix you have applied to resolve this issue?

Re: strange error with dbutils.notebook.run(...)

Pramod_Data_Eng — Mon, 31 Jan 2022 22:27:50 GMT

Hello Support,

Still i have similar kind of problem.

Error: com.databricks.NotebookExecutionException: FAILED

Cluster Mode : Standard

I tried fetching the data from the event hubs. Previously , it worked good on one databricks instance. with runtime( 7.6 (includes Apache Spark 3.0.1, Scala 2.12)

spark.executor.extraJavaOptions -Dlog4j2.formatMsgNoLookups=true

spark.driver.extraJavaOptions -Dlog4j2.formatMsgNoLookups=true

spark.databricks.delta.preview.enabled true)

Later i tried running on the newly created instance with runtime (7.5 (includes Apache Spark 3.0.1, Scala 2.12)

spark.executor.extraJavaOptions -Dlog4j2.formatMsgNoLookups=true

spark.driver.extraJavaOptions -Dlog4j2.formatMsgNoLookups=true

spark.databricks.delta.preview.enabled true)

Please, help me resolve the issue.

Re: strange error with dbutils.notebook.run(...)

Pramod_Data_Eng — Mon, 31 Jan 2022 22:30:00 GMT

attached the error file.

Re: strange error with dbutils.notebook.run(...)

User16753724663 — Wed, 02 Feb 2022 07:22:56 GMT

Hi @Florent POUSSEROT

Apologies for the delay. Could you please confirm if you are still facing the issue?

Re: strange error with dbutils.notebook.run(...)

florent — Wed, 02 Feb 2022 20:33:13 GMT

Hi, for the moment, it's ok