Hi @werners , thanks for your response.

As a beginner , I would like to use pyspark.pandas as a plug and play. (by converting my classic pandas code to pyspark.pandas ).

Would you know why I am getting the error (mentioned in the question)?

It's really peculiar that it happens only with larger datasets..

Do you recommend I raise an issue with Databricks ?