@NandiniN 

Did you try with a multiple worker cluster? Which Runtime with which spark version did you use?

Maybe it would be good to test with Runtime 13.3, then we would know that it was fixed in the meantime.

I found this on StackOverflow. Seems someone had a similar problem: https://stackoverflow.com/questions/55860388/pyspark-dataframe-orderby-partition-level-or-overall

There is also a very old HIVE bug ticket that was never resolved. Not sure, if it could be connected:

https://issues.apache.org/jira/browse/HIVE-10417