RuntimeError: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.

Nikhil3107
New Contributor III

I created the following model:

 

image 

which calls get_identifier_information() which is as follows:

 

image 

This is how I log the model

 

image 

And this is the error I am running into:

 

RuntimeError: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.