topic Re: RuntimeError: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063. in Warehousing & Analytics

RuntimeError: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.

Nikhil3107 — Thu, 20 Mar 2025 17:07:53 GMT

I created the following model:

which calls get_identifier_information() which is as follows:

This is how I log the model

And this is the error I am running into:

RuntimeError: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.

Re: RuntimeError: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.

Anonymous — Fri, 16 Jun 2023 06:01:38 GMT

Hi @Nikhil Gajghate

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question. Thanks.