Hi,

We are planning to re-write our application ( which was originally running in R) in python. We chose to use Polars as they seems to be faster than pandas. We have functions written in R which we are planning to convert to Python.

However in one of the recent trainings by databrcks that I attended, I got the idea that when using python functions, Spark does not distribute the work to worker node and it will be run on driver. I understand that is the case with pandas. So would like to know if running functions written in Polars also have the same disadvantage.

what are the alternatives to this? 

our data is kind of time series..that is, the order of the rows is important. so when we tried converting the code to sparkR the results were not as expected.

Is there an recommended approach for this type of data?

Warm Regards

Manjusha