Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-01-2025 05:11 AM - edited 05-01-2025 05:12 AM
You could use
mode(defaultPERMISSIVEallows a mode for dealing with corrupt records during parsing.PERMISSIVE: when it meets a corrupted record, puts the malformed string into a field configured bycolumnNameOfCorruptRecord, and sets malformed fields to null. To keep corrupt records, you can set a string type field namedcolumnNameOfCorruptRecordin an user-defined schema. If a schema does not have the field, it drops corrupt records during parsing. When inferring a schema, it implicitly adds acolumnNameOfCorruptRecordfield in an output schema.
columnNameOfCorruptRecord(default is the value specified inspark.sql.columnNameOfCorruptRecordallows renaming the new field having malformed string created byPERMISSIVEmode. This overridesspark.sql.columnNameOfCorruptRecord.
Doc - https://docs.databricks.com/aws/en/sql/language-manual/functions/from_json