topic Pyspark serialization in Data Engineering

Pyspark serialization

yusufd — Fri, 21 Jun 2024 11:50:53 GMT

Hi,

I was looking for comprehensive documentation on implementing serialization in pyspark, most of the places I have seen is all about serialization with scala. Could you point out where I can get a detailed explanation on it?

Re: Pyspark serialization

yusufd — Wed, 26 Jun 2024 07:37:20 GMT

This is awesome. Thank you for replying.

I want to ask one more thing before we close this, in Scala-spark java serialization is default and additionally we have kryo serialization as well which is better. So these are not applicable in pyspark if i get correctly. Kindly confirm.

Re: Pyspark serialization

yusufd — Wed, 26 Jun 2024 09:50:14 GMT

This is awesome. Thank you for replying.

I want to ask one more thing before we close this, in Scala-spark java serialization is default and additionally we have kryo serialization as well which is better. So, can we use them in pyspark as well?

Another important thing, the code below creates a sparkcontext on local, that doesnt work on databricks. When I try to change the sparkcontext arguments, i get an error , attached screenshot, how can we resolve this, ultimately i dont want to run spark locally, but on databricks. Would appreciate if you answer this.

Thanks for the support.

Re: Pyspark serialization

yusufd — Mon, 01 Jul 2024 11:27:26 GMT

@Retired_mod Could you clarify on my query? Eagerly awaiting response.

Re: Pyspark serialization

yusufd — Mon, 01 Jul 2024 13:05:46 GMT

Thank you @Retired_mod for the prompt reply. This clears the things and also distinguishes between spark-scala and pyspark. Appreciate your explanation. Will apply this and also share any findings based on this which will help the community!

Re: Pyspark serialization

yusufd — Mon, 01 Jul 2024 13:06:58 GMT

This is great to know!

Thank you for the explanation.