strange error with dbutils.notebook.run(...)

[Mirko]

The situation is as following: i have a sheduled job, which uses dbutils.notebook.run(path,timeout) . During the last week everything worked smooth. During the weekend the job began to fail, at the dbutils.notebook.run(path,timeout) command. I get the following error:

com.databricks.WorkflowException: com.databricks.NotebookExecutionException: FAILED: assertion failed: Attempted to set keys (credentials) in the extraContext, but these keys were not in the set of valid keys: {commandResultJsonMaxBytes, displayRowLimitV2, notebook_path, thresholdForStoringInDbfs, enableStoringResultsInDbfs, api_url, aclPathOfAclRoot, displayByteLimitV2}

to be honest, i am not 100% sure if my hoster changed anything in the databricks config during the weekend, but maybe somebody knows which situation would result in such an error.

[User16753724663 · Accepted solution]

Hi @Mirko Ludewig​ 

This issue seems to be with High Concurrency cluster. Could you please confirm if you are using high concurrency cluster?.

[ShaikMaheer]

Yes this issue faced in high concurrency Cluster when pass through is enabled in Azure data bricks. Any insights about this issue will be helpful?

[Mirko]

@DARSHAN BARGAL​ @Shaik Maheer​  first, exsuce me for the delayed answer. I Asked my admin to create a high con cluster for me as well, because i wanted to check both cases. In fact i used a standard cluster all the time. Strangely today everything works fine. With both high con and standaard clusters. I guess that is a good thing

[User16753724663]

Hi @Mirko Ludewig​ @Shaik Maheer​ ,

Thank you for the update. We applied the fix from the backend to resolve this issue. We apologies for the inconvenience caused here. I hope everything is working fine now at your end.

[KC_Databricks]

@DARSHAN BARGAL​ 

I am facing a similar error at a client location today (19Jan2022). This is on a pass-thru high con cluster. What was the fix you have applied to resolve this issue?

[Pramod_Data_Eng]

Hello Support,

Still i have similar kind of problem.

Error: com.databricks.NotebookExecutionException: FAILED

Cluster Mode : Standard

I tried fetching the data from the event hubs. Previously , it worked good on one databricks instance. with runtime( 7.6 (includes Apache Spark 3.0.1, Scala 2.12)

spark.executor.extraJavaOptions -Dlog4j2.formatMsgNoLookups=true

spark.driver.extraJavaOptions -Dlog4j2.formatMsgNoLookups=true

spark.databricks.delta.preview.enabled true)

Later i tried running on the newly created instance with runtime (7.5 (includes Apache Spark 3.0.1, Scala 2.12)

spark.executor.extraJavaOptions -Dlog4j2.formatMsgNoLookups=true

spark.driver.extraJavaOptions -Dlog4j2.formatMsgNoLookups=true

spark.databricks.delta.preview.enabled true)

Please, help me resolve the issue.

[Pramod_Data_Eng]

attached the error file.

[Mirko]

@DARSHAN BARGAL​  thank you for the explanation. It is good to know that i can recieve help quickly here!

[florent]

Hi @DARSHAN BARGAL​ ,

we have the same problem and it is still present at 11:31 utc

Cluster : standard

Passtrough :on

Runtime : 9.1 ml

region : azure francecentral

[Mirko]

@Florent POUSSEROT​  my check was on runtime 10.1

[User16753724663]

Hi @Florent POUSSEROT​ 

Apologies for the delay. Could you please confirm if you are still facing the issue?

[florent]

Hi, for the moment, it's ok