06-16-2021 05:39 AM - last edited on 03-20-2025 10:19 AM by Advika
While trying to read 100GB of csv.gz file from Spark which is taking forever to read, what are best options to read this file faster?
06-17-2021 04:06 PM
Part of the problem here is that .gz files are not splittable. If you have 1 huge 100GB .gz file, it can only be processed by one task. Can you change your input to use a splittable compression like .bz2? it'll work much better.
