@Suteja Kanuri​ 

Thank for the response.

So, given that my DataBricks cluster configuration is 2-8 worker nodes, the driver nodes will allocate the data to be processed in parallel to 4 worker nodes because I have specified `4` in the Pool object? What happens when the `num_processes` I have specified is greater than or less than the available worker nodes.

Would you mind sharing an example of how I'd implement the parallel processing of applying the function along the row axis of a pandas using spark and RDDs?