RuntimeError: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.

[Nikhil3107]

I created the following model:

 

which calls get_identifier_information() which is as follows:

 

This is how I log the model

 

And this is the error I am running into:

RuntimeError: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.

[Anonymous]

Hi @Nikhil Gajghate​ 

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question. Thanks.

[Kaniz_Fatma · Accepted solution]

Hi @Nikhil Gajghate​, The error message you encountered indicates that you are attempting to reference the SparkContext object from a broadcast variable, action, or transformation in your code running on worker nodes. However, the SparkContext object can only be used on the driver node and is not accessible within code executed on worker nodes.