We have customers that read millions of files per hour+ using Databricks Auto Loader. For high-volume use cases, we recommend enabling file notification mode, which, instead of continuously performing list operations on the filesystem, uses cloud native offerings to "push" new file locations to Auto Loader.
For handling bad records, we have the rescuedDataColumn and for JSON files in particular, columnNameOfCorruptRecord. You can use these in tandem with some additional filtering logic to quarantine bad records away.