topic Error: Credential size is more than configured size limit. As a result credential passthrough won't work for this notebook run. in Data Engineering

Error: Credential size is more than configured size limit. As a result credential passthrough won't work for this notebook run.

carlosancassani — Thu, 13 Oct 2022 10:16:31 GMT

I get this error when trying to execute parallel slave notebook from a Pyspark "master notebook".

note 1: I use same class, functions, cluster, credential for another use case of parallel notebook in the same databricks instance and it works fine.

note 2: the command works fine if the "master notebook" is launched from a Job, while it returns the error above when the notebook is launched manually.

So far couldn't find similar errors in docs or forums.

Re: Error: Credential size is more than configured size limit. As a result credential passthrough won't work for this notebook run.

Hubert-Dudek — Tue, 18 Oct 2022 12:05:41 GMT

Mabe, it is the issue:

"Within PySpark, there is a limit on the size of the Python UDFs you can construct since large UDFs are sent as broadcast variables."

Re: Error: Credential size is more than configured size limit. As a result credential passthrough won't work for this notebook run.

carlosancassani — Tue, 18 Oct 2022 13:19:16 GMT

Hi,

I add a note: there are no UDF's in the "master notebook" nor the "slave notebooks"; the command works fine if the "master notebook" is launched from a Job, while it returns the error above when the notebook is launched manually.

Re: Error: Credential size is more than configured size limit. As a result credential passthrough won't work for this notebook run.

Anonymous — Mon, 21 Nov 2022 07:01:21 GMT

Hi @carlosancassani

Hope all is well!

Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!