Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-24-2017 08:07 AM
First you are bouncing between the RDD and DataFrames API. If you start with a SparkSession object from the DF API instead you can make the call
spark.read.option("header", "true").csv(path)then you don't have to split the file, convert from RDD to DF and the first column will be read as a header instead of as data.
For more information you can also look at the ~20 other options available to the DataFrameReader (spark.read)