why spark very slow with large number of dataframe columns
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-06-2018 08:37 AM
scala Spark App: I have a dataset of 130x14000. I read from a parquet file with SparkSession. Then used for Spark ML Random Forest model (using pipeline). It takes 7 hours to complete! for reading the parquet file takes about 1 minute. If I implement it, it takes less than 3 seconds to complete (from reading the data and completing the modeling). What could be the problem with Scala , Spark Dataframe implementation?
Labels:
- Labels:
-
Scala