NandiniN
Databricks Employee
Databricks Employee

You could use

  • mode (default PERMISSIVE allows a mode for dealing with corrupt records during parsing.
    • PERMISSIVE: when it meets a corrupted record, puts the malformed string into a field configured by columnNameOfCorruptRecord, and sets malformed fields to null. To keep corrupt records, you can set a string type field named columnNameOfCorruptRecord in an user-defined schema. If a schema does not have the field, it drops corrupt records during parsing. When inferring a schema, it implicitly adds a columnNameOfCorruptRecord field in an output schema.
  • columnNameOfCorruptRecord (default is the value specified in spark.sql.columnNameOfCorruptRecord allows renaming the new field having malformed string created by PERMISSIVE mode. This overrides spark.sql.columnNameOfCorruptRecord.

Doc - https://docs.databricks.com/aws/en/sql/language-manual/functions/from_json