Why Entire Files Go to badRecordsPath
When you enable badRecordsPath in Autoloader or in Spark’s file readers (with formats like CSV/JSON), here’s what happens:
Spark expects each data file to be internally well-formed with respect to the declared s...
Hi @shan-databricks , Try with below option
df = ( spark.readStream .format("cloudFiles")
.option("cloudFiles.format", "csv")
.option("badRecordsPath", "/mnt/my-bad-records")
#.option("mode", "PERMISSIVE") # Do NOT set this!
.schema(my_schema...
