DQ-Quality Check- what are the best method to validate the two parquet files .
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-15-2024 10:56 PM
DQ-Quality Check. we have to validate the data between landing data and bronze data with data quality . below are the data quality checks.
1. find the counts between the 2 files. if it is matched then go for 2 point.
2. if counts are matched, then validate the data row by row as per keys . if keys are matched, then validate the data between the other columns. if the columns are not matched then store in error log file.
what is best methodology we can go for in pyspark(databricks).
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-21-2024 06:47 AM
what you are looking for is except and exceptAll.
f.e. df1.except(df2)
it returns the data of df1 that has no match in df2.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-21-2024 08:18 AM
Thanks Werners. will it provide the good performance?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-22-2024 12:12 AM
It does use spark. But of course it is an expensive operation as all records are compared.
In my experience the performance is reasonable.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-22-2024 12:34 AM
Try with this , this is for second point if first points already matches .
# Define key columns
key_columns = ["key_column1", "key_column2"] # Adjust according to your data schema
# Perform an outer join to find mismatches
joined_df = landing_df.alias("landing").join(
bronze_df.alias("bronze"),
on=key_columns,
how="outer"
)
