02-05-2023 05:06 AM
Hi, are all the rows missing the column or a few?
Also, you can you filter out bad rows using expectations , or your DQ framework and backfill it once the corrected records arrive or you can use
.option("overwriteSchema", "true")
