i am using rdd to parallelize a function, in this function i format the record i want to save, how can i store from this function the record with a dataframe? because every time i use spark..... an error is generated Caused by: org.apache.spark.api.python.PythonException: 'RuntimeError: SparkContext should only be created and accessed on the driver.'
How can I somehow run spark. something in a worker?
code - images