Lakshay
Databricks Employee
Databricks Employee

It looks like the 3m records are coming from one file. And to process these records, you might need more cores in your cluster.