Data comparison

[Frustrated_DE]

Hi,

   Are there any tools within Databricks for large volume data comparisons, I appreciate there's methods for dataframe comparisons for unit testing (assertDataFrameEqual) but it is my understanding these are for testing transformations on smallish data. I have sizeable datasets that I would like to compare to ensure the values are equal before starting another pipeline and hoping to find an efficient way of undertaking this exercise. Any thoughts appreciated.

                    Thanks

[szymon_dybczak]

Hi @Frustrated_DE ,

I don't know if that's what you're looking for, but maybe you can use set operators to compare dataframes ( intersect, except). If both dataframes have the same schema, except operator gives you difference between two sets of data. Intersect will return data that is common in both datasets.

Set operators are pretty handy when it comes to data quality validation.

[Frustrated_DE]

Thanks Szymon, I will give these a try!