Brian's answer above is probably the simplest. Here are some other options in case line.seperator doesn't do the trick:

Option: Custom CSV Reader

Modify the CSV reader and upload it as a library. Here are the steps:

• Fork the current CSV reader from https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/d...
• Merge the pull request into your fork
• Change the package name
• Load it as a JAR library
• Access your custom reader using spark.read.format("com.example.spark.datasources.csv.CSVFileFormat").load(filename)

Option: Fix the original file...

Create a program that reads the file in as a byte stream, repairs the bytes, and writes out a new, repaired file. It's easiest if you do it without parallelizing, but you could parallelize if you wanted by breaking the file into 1 gb chunks and seeking ahead to the relevant section to process those in parallel.