Error: Credential size is more than configured size limit. As a result credential passthrough won't work for this notebook run.

carlosancassani
New Contributor III

‎10-13-2022 03:16 AM

I get this error when trying to execute parallel slave notebook from a Pyspark "master notebook".

note 1: I use same class, functions, cluster, credential for another use case of parallel notebook in the same databricks instance and it works fine.

note 2: the command works fine if the "master notebook" is launched from a Job, while it returns the error above when the notebook is launched manually. 

image 

So far couldn't find similar errors in docs or forums.

Preview file
81 KB

Hubert-Dudek
Databricks MVP

‎10-18-2022 05:05 AM

Mabe, it is the issue:

"Within PySpark, there is a limit on the size of the Python UDFs you can construct since large UDFs are sent as broadcast variables."


My blog: https://databrickster.medium.com/

carlosancassani
New Contributor III

‎10-18-2022 06:19 AM

Hi,

I add a note: there are no UDF's in the "master notebook" nor the "slave notebooks"; the command works fine if the "master notebook" is launched from a Job, while it returns the error above when the notebook is launched manually.

Anonymous
Not applicable

‎11-20-2022 11:01 PM

Hi @carlosancassani​ 

Hope all is well!

Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

0 Kudos