Trigger bad records in databricks

bjn
New Contributor III

‎05-07-2025 01:37 AM

I use bad records while reading a csv as follows:

df = spark.read.format("csv")
            .schema(schema)
            .option("badRecordsPath", bad_records_path)
 
Since bad records are not written immediately, I want to know how can trigger the write of them efficiently. 
 
Currently, I use df.collect() to trigger the bad records write which cause massive overhead and even out of memory problems, which is not acceptable. 
0 Kudos