https://community.databricks.com/t5/warehousing-analytics/reading-bulk-csv-files-from-spark/m-p/24045#M595 <P>While trying to read 100GB of csv.gz file from Spark which is taking forever to read, what are best options to read this file faster?</P> Thu, 20 Mar 2025 17:19:55 GMT Srikanth_Gupta_ 2025-03-20T17:19:55Z https://community.databricks.com/t5/warehousing-analytics/reading-bulk-csv-files-from-spark/m-p/24045#M595 <P>While trying to read 100GB of csv.gz file from Spark which is taking forever to read, what are best options to read this file faster?</P> Thu, 20 Mar 2025 17:19:55 GMT https://community.databricks.com/t5/warehousing-analytics/reading-bulk-csv-files-from-spark/m-p/24045#M595 Srikanth_Gupta_ 2025-03-20T17:19:55Z https://community.databricks.com/t5/warehousing-analytics/reading-bulk-csv-files-from-spark/m-p/24046#M596 <P>Part of the problem here is that .gz files are not splittable. If you have 1 huge 100GB .gz file, it can only be processed by one task. Can you change your input to use a splittable compression like .bz2? it'll work much better.</P> Thu, 17 Jun 2021 23:06:32 GMT https://community.databricks.com/t5/warehousing-analytics/reading-bulk-csv-files-from-spark/m-p/24046#M596 sean_owen 2021-06-17T23:06:32Z