Thanks @Kaniz Fatma​, this has solved half of the problem, the other half is that I need to Transpose the pyspark dataframe. Any help on this one?

Riz