RuntimeError: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.

Nikhil3107
New Contributor III

‎06-07-2023 08:35 AM - last edited on ‎03-20-2025 10:07 AM by Community Manager

I created the following model:

 

image 

which calls get_identifier_information() which is as follows:

 

image 

This is how I log the model

 

image 

And this is the error I am running into:

 

RuntimeError: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.

 

 

 

Preview file
37 KB
Preview file
32 KB
Preview file
66 KB

Anonymous
Not applicable

‎06-15-2023 11:01 PM

Hi @Nikhil Gajghate​ 

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question. Thanks.