topic Re: Comparing two dataframes in Data Engineering

Comparing two dataframes

SiddarthaPaturu — Thu, 31 Mar 2016 20:53:51 GMT

How can we compare two data frames using pyspark

I need to validate my output with another dataset

Re: Comparing two dataframes

girivaratharaja — Thu, 31 Mar 2016 22:22:46 GMT

>>> df1.subtract(df2)

As per API Docs, it returns a new DataFrame containing rows in this frame but not in another frame.

This is equivalent to EXCEPT in SQL.

https://spark.apache.org/docs/1.3.0/api/python/pyspark.sql.html?highlight=dataframe#pyspark.sql.DataFrame.subtract

Re: Comparing two dataframes

SiddarthaPaturu — Mon, 04 Apr 2016 13:38:53 GMT

Its giving only the rows which or not in other data frame, Is there anything that validate all the column values in both the df

Re: Comparing two dataframes

girivaratharaja — Mon, 04 Apr 2016 17:20:22 GMT

@Siddartha Paturu I If that is the case, I would recommend to do Join between two dataframes and then compare it for all columns

Re: Comparing two dataframes

SiddarthaPaturu — Tue, 05 Apr 2016 15:36:14 GMT

how can we compare the columns ?

Re: Comparing two dataframes

jagannathsahoo — Wed, 20 Jul 2016 15:21:40 GMT

recently I am also stuck with this situation. can somebody help me with how to compare columns in this scenario. @Siddartha Paturu please help me out with this if already found the solution. Thanks in advance.

Re: Comparing two dataframes

ShashishekharDe — Tue, 20 Sep 2016 22:29:42 GMT

I am stuck with the same issue.. Any new updates on this?

Is there any solution to this problem??

Re: Comparing two dataframes

amandaphy — Sat, 24 Sep 2016 08:27:24 GMT

Try using

all.equal

function.

It does not sort the dataframes but it checks each cell in

data frame

against the same cell in another one. You can also use

identical()

function.

I would like to share a link which may help to solve your problem https://goo.gl/pgLaEd

Re: Comparing two dataframes

sbharti — Thu, 28 Jun 2018 13:53:44 GMT

I think the best bet in such a case is to take inner join (equivalent to intersection) by putting a condition on those columns which necessarily need to have same value in both dataframes. For example,

let df1 and df2 are two dataframes. df1 has column (A,B,C) and df2 has columns (D,C,B), then you can create a new dataframe which would be the intersection of df1 and df2 conditioned on column B and C.

df3 = df1.join(df2, [df1.B == df2.B , df1.C == df2.C], how = 'inner' )

df3 will contain only those rows where the above condition is satisfied from df1 and df2.