Lakshay
Databricks Employee
Databricks Employee

Hi @Ankit Gangwal​ , The problem with the zip files is that they are not splittable and only use one core to process. It is better to change the compression format to snappy as it is splittable and will allow spark to distribute the workload over the cluster.

Ref link:- https://www.linkedin.com/pulse/apache-spark-optimizations-compression-deepak-rajak

View solution in original post