NandiniN
Databricks Employee
Databricks Employee

The orderBy function in PySpark is expected to perform a global sort, which involves shuffling the data across partitions to ensure that the entire DataFrame is sorted. This is different from sortWithinPartitions, which only sorts data within each partition.

Let me try your program and understand the results further.