I actually found an option that could solve the newline issue I mentioned in my previous post:

setting spark.sql.csv.parser.columnPruning.enabled to false with

spark.conf.set("spark.sql.csv.parser.columnPruning.enabled", False)

will consider malformed rows also partial rows (i.e, with a wrong number of separators), meaning that missing fields are NOT automatically filled with nulls (triggering schema errors only in case nullability is not enforced).

The default = True has been set from spark 2.4 (Spark SQL Upgrading Guide - Spark 2.4.0 Documentation (apache.org)) , but I can't really find any detailed explanation of the other consequences.