PySparkRuntimeError: [CONTEXT_ONLY_VALID_ON_DRIVER] It appears that you are attempting to reference

pinaki1
New Contributor III

‎08-27-2024 06:06 AM

Getting The above error for this line
result_df.rdd.foreachPartition(self.process_partition)

0 Kudos

Pradeep54
Databricks Employee
Databricks Employee

‎09-28-2024 11:37 AM

The error message "CONTEXT_ONLY_VALID_ON_DRIVER" indicates that you are attempting to reference SparkContext from a broadcast variable, action, or transformation. SparkContext can only be used on the driver, not in code that runs on workers. This is a common issue in PySpark where certain operations are mistakenly attempted on worker nodes instead of the driver node

0 Kudos