Pyspark serialization

[yusufd]

Hi,

I was looking for comprehensive documentation on implementing serialization in pyspark, most of the places I have seen is all about serialization with scala. Could you point out where I can get a detailed explanation on it?

[yusufd]

This is awesome. Thank you for replying. 

I want to ask one more thing before we close this, in Scala-spark java serialization is default and additionally we have kryo serialization as well which is better. So these are not applicable in pyspark if i get correctly. Kindly confirm.

[yusufd]

This is great to know!

Thank you for the explanation.

[yusufd]

This is awesome. Thank you for replying. 

I want to ask one more thing before we close this, in Scala-spark java serialization is default and additionally we have kryo serialization as well which is better. So, can we use them in pyspark as well?

Another important thing, the code below creates a sparkcontext on local, that doesnt work on databricks. When I try to change the sparkcontext arguments, i get an error , attached screenshot, how can we resolve this, ultimately i dont want to run spark locally, but on databricks. Would appreciate if you answer this.

Thanks for the support.

[yusufd]

@Retired_mod Could you clarify on my query? Eagerly awaiting response.

[yusufd]

Thank you @Retired_mod  for the prompt reply. This clears the things and also distinguishes between spark-scala and pyspark. Appreciate your explanation. Will apply this and also share any findings based on this which will help the community!