Hi @bjn ,

If I underestand you correctly

To efficiently trigger the writing of bad records captured via .option("badRecordsPath", ...) without causing memory overhead or driver-side issues, the best option is:

df.write.format("noop").mode("overwrite").save()

This forces Spark to fully evaluate the DataFrame (including the detection and writing of bad records), but without writing any actual output data. It’s more efficient than using .collect() or even .count(), since it avoids aggregations and doesn’t load data into the driver.


Hope this helps, 🙂

Isi