Error: Credential size is more than configured size limit. As a result credential passthrough won't work for this notebook run.

[carlosancassani]

I get this error when trying to execute parallel slave notebook from a Pyspark "master notebook".

note 1: I use same class, functions, cluster, credential for another use case of parallel notebook in the same databricks instance and it works fine.

note 2: the command works fine if the "master notebook" is launched from a Job, while it returns the error above when the notebook is launched manually. 

 

So far couldn't find similar errors in docs or forums.

[Hubert-Dudek]

Mabe, it is the issue:

"Within PySpark, there is a limit on the size of the Python UDFs you can construct since large UDFs are sent as broadcast variables."

[carlosancassani]

Hi,

I add a note: there are no UDF's in the "master notebook" nor the "slave notebooks"; the command works fine if the "master notebook" is launched from a Job, while it returns the error above when the notebook is launched manually.

[Anonymous]

Hi @carlosancassani​ 

Hope all is well!

Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!