https://community.databricks.com/t5/data-engineering/udf-function-while-registering-picklingerror/m-p/15760#M10036 <P>PicklingError: Could not serialize object: Exception: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers.</P><P></P><P>I am trying to write a function in Azure databricks. I would like to spark.sql inside the function. But it looks like I cannot use it with worker nodes.</P><P></P><OL><LI>def SEL_ID(value, index):</LI><LI> # some processing on value here</LI><LI> ans = spark.sql("SELECT id FROM table WHERE bin = index")</LI><LI> return ans</LI><LI>spark.udf.register("SEL_ID", SEL_ID)</LI></OL><P></P><P>I am getting the following error:</P><P></P><P>PicklingError: Could not serialize object: Exception: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.</P><P></P><P>Is there any way I can overcome this? I am using the above function to select from another table.</P> Tue, 20 Dec 2022 12:30:31 GMT databricks_amit 2022-12-20T12:30:31Z https://community.databricks.com/t5/data-engineering/udf-function-while-registering-picklingerror/m-p/15760#M10036 <P>PicklingError: Could not serialize object: Exception: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers.</P><P></P><P>I am trying to write a function in Azure databricks. I would like to spark.sql inside the function. But it looks like I cannot use it with worker nodes.</P><P></P><OL><LI>def SEL_ID(value, index):</LI><LI> # some processing on value here</LI><LI> ans = spark.sql("SELECT id FROM table WHERE bin = index")</LI><LI> return ans</LI><LI>spark.udf.register("SEL_ID", SEL_ID)</LI></OL><P></P><P>I am getting the following error:</P><P></P><P>PicklingError: Could not serialize object: Exception: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.</P><P></P><P>Is there any way I can overcome this? I am using the above function to select from another table.</P> Tue, 20 Dec 2022 12:30:31 GMT https://community.databricks.com/t5/data-engineering/udf-function-while-registering-picklingerror/m-p/15760#M10036 databricks_amit 2022-12-20T12:30:31Z