Context: I am using pyspark.pandas in a Databricks jupyter notebook and doing some text manipulation within the dataframe..

pyspark.pandas is the Pandas API on Spark and can be used exactly the same as usual Pandas

Error: PicklingError: Could not serialize object: TypeError: cannot pickle '_thread.RLock' object

Some clues that can help you understand the error:

I do not get any error if I run my script on:

• 300 rows of data.
• 600 rows of data (created by replicating the original 300 x2)

I get an error if I run my script on :

• On 3000 rows of data (created by replicating the original 300 x10)
• On 3000 unique rows (not related to the original 300 rows)

This makes me think the error is not code specific rather databricks/pyspark.pandas might have an intricacy / limitation / bug which happens with higher number of rows of data(3000 in my case)

Can somebody please explain why I am getting this error and how to resolve it?

Would appreciate if we stick to pyspark.pandas and not go into alternatives that suggest using spark sql..

If you want to look at the full stack trace for the error , you may check the code snippet you see in this question which I have asked.