DQ-Quality Check. we have to validate the data between landing data and bronze data with data quality . below are the data quality checks.
1. find the counts between the 2 files. if it is matched then go for 2 point.
2. if counts are matched, then validate the data row by row as per keys . if keys are matched, then validate the data between the other columns. if the columns are not matched then store in error log file.
what is best methodology we can go for in pyspark(databricks).