Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-16-2021 10:01 AM
you can save corrupted records to separate file:
.option("badRecordsPath", "/tmp/badRecordsPath")allow spark to process corrupted row:
.option("mode", "PERMISSIVE")you can also create special column for corrupted records:
df = spark.read.csv('/tmp/inputFile.csv', header=True, schema=dataSchema, enforceSchema=True,
columnNameOfCorruptRecord='CORRUPTED')
My blog: https://databrickster.medium.com/