https://community.databricks.com/t5/data-engineering/pysparkruntimeerror-context-only-valid-on-driver-it-appears-that/m-p/84392#M37187 <P>Getting The above error for this line<BR /><STRONG><EM><SPAN class="">result_df</SPAN><SPAN class="">.</SPAN><SPAN class="">rdd</SPAN><SPAN class="">.</SPAN><SPAN class="">foreachPartition</SPAN><SPAN class="">(</SPAN><SPAN class="">self</SPAN><SPAN class="">.</SPAN><SPAN class="">process_partition</SPAN><SPAN class="">)</SPAN></EM></STRONG></P> Tue, 27 Aug 2024 13:06:40 GMT pinaki1 2024-08-27T13:06:40Z https://community.databricks.com/t5/data-engineering/pysparkruntimeerror-context-only-valid-on-driver-it-appears-that/m-p/84392#M37187 <P>Getting The above error for this line<BR /><STRONG><EM><SPAN class="">result_df</SPAN><SPAN class="">.</SPAN><SPAN class="">rdd</SPAN><SPAN class="">.</SPAN><SPAN class="">foreachPartition</SPAN><SPAN class="">(</SPAN><SPAN class="">self</SPAN><SPAN class="">.</SPAN><SPAN class="">process_partition</SPAN><SPAN class="">)</SPAN></EM></STRONG></P> Tue, 27 Aug 2024 13:06:40 GMT https://community.databricks.com/t5/data-engineering/pysparkruntimeerror-context-only-valid-on-driver-it-appears-that/m-p/84392#M37187 pinaki1 2024-08-27T13:06:40Z https://community.databricks.com/t5/data-engineering/pysparkruntimeerror-context-only-valid-on-driver-it-appears-that/m-p/92179#M38379 <P>The error message "CONTEXT_ONLY_VALID_ON_DRIVER" indicates that you are attempting to reference <CODE>SparkContext</CODE> from a broadcast variable, action, or transformation. <CODE>SparkContext</CODE> can only be used on the driver, not in code that runs on workers. This is a common issue in PySpark where certain operations are mistakenly attempted on worker nodes instead of the driver node</P> Sat, 28 Sep 2024 18:37:35 GMT https://community.databricks.com/t5/data-engineering/pysparkruntimeerror-context-only-valid-on-driver-it-appears-that/m-p/92179#M38379 Pradeep54 2024-09-28T18:37:35Z