Thank you very much, I ill try to do that as it seems that that is the problem! Nevertheless, I managed to save the dataframe into CSV and from there to transform it to pandas (it did not work from me directly from spark df to pandas). Pandas works great with this dataset as it is not quite big. However, I am aware that it is not suitable for big data. So for big data, next time, I will try to connect to existing spark cluster.