@Radu Gheorghiu​ ,

• You could also try to prepare a sample from your JSON like 1 MB and check is it processed correctly. It will be easier to debug.
• some ideas:
• please add .option("badRecordsPath", "/tmp/badRecordsPath") and/or mode PERMISSIVE
• I know it is pretty complex, but it could be good to define read schema to avoid second reading.
• if you have a single JSON object per line, you could read it as a text file with ( read.textFile), do partitioning, etc., and then use JSON functions to process data