- 1369 Views
- 1 replies
- 0 kudos
Resolved! what is catalyst in spark
- 1369 Views
- 1 replies
- 0 kudos
- 0 kudos
Spark SQL was designed with an optimizer called Catalyst based on the functional programming of Scala. Its two main purposes are: first, to add new optimization techniques to solve some problems with “big data” and second, to allow developers to expa...
- 0 kudos
- 2159 Views
- 1 replies
- 1 kudos
Reading bulk CSV files from Spark
While trying to read 100GB of csv.gz file from Spark which is taking forever to read, what are best options to read this file faster?
- 2159 Views
- 1 replies
- 1 kudos
- 1 kudos
Part of the problem here is that .gz files are not splittable. If you have 1 huge 100GB .gz file, it can only be processed by one task. Can you change your input to use a splittable compression like .bz2? it'll work much better.
- 1 kudos