Fair enough... let's try another solution then.

One option would be to use sc.wholeTextFiles(..) and simply remove the first line of the file. Personally, I don't like this because each file is forced into one partition and if it is a really large file, then I can have issues with running out of memory.

A second (and third) option revolves around how do I filter() out row #1 which contains a header and in this case would require having special knowledge about the data.

In your example above, the first column has values like E01, E02, E03, etc. If we can say definitively that the value "ID" is invalid then we can simply exclude any record with that value. You would do this just after you split the CSV file up, or more specifically, after the call to xxx.map(). It has the disadvantage that you are only checking one column of data and may not be as accurate as you need.

A third option, possibly easier to implement, is to filter() out any line of text that is a specific value - that is the entire line matches your column headers. In this case, the filter accepts any line that does not equal "ID,Employee_name". You would do this just after the call to sc.textFile() and before xxx.map().

And if you really want to get tricky, you can read in just the first line of your file to determine what the header is, and then use it in the filter with option three above.