PySparkRuntimeError: [CONTEXT_ONLY_VALID_ON_DRIVER] It appears that you are attempting to reference

[pinaki1]

Getting The above error for this line
result_df.rdd.foreachPartition(self.process_partition)

[User16753724828]

The error message "CONTEXT_ONLY_VALID_ON_DRIVER" indicates that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that runs on workers. This is a common issue in PySpark where certain operations are mistakenly attempted on worker nodes instead of the driver node