PySparkRuntimeError: [CONTEXT_ONLY_VALID_ON_DRIVER] It appears that you are attempting to reference
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-27-2024 06:06 AM
Getting The above error for this line
result_df.rdd.foreachPartition(self.process_partition)
1 REPLY 1
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2024 11:37 AM
The error message "CONTEXT_ONLY_VALID_ON_DRIVER" indicates that you are attempting to reference SparkContext
from a broadcast variable, action, or transformation. SparkContext
can only be used on the driver, not in code that runs on workers. This is a common issue in PySpark where certain operations are mistakenly attempted on worker nodes instead of the driver node