This is awesome. Thank you for replying. 

I want to ask one more thing before we close this, in Scala-spark java serialization is default and additionally we have kryo serialization as well which is better. So these are not applicable in pyspark if i get correctly. Kindly confirm.