Hubert-Dudek
Databricks MVP

you can save corrupted records to separate file:

.option("badRecordsPath", "/tmp/badRecordsPath")

allow spark to process corrupted row:

.option("mode", "PERMISSIVE")

you can also create special column for corrupted records:

df = spark.read.csv('/tmp/inputFile.csv', header=True, schema=dataSchema, enforceSchema=True, 
 
columnNameOfCorruptRecord='CORRUPTED')


My blog: https://databrickster.medium.com/