topic Re: Conditionally create a dataframe in Data Engineering

Conditionally create a dataframe

Braxx — Thu, 04 Nov 2021 09:12:27 GMT

I would like to implement a simple logic:

if Df1 is empty return Df2 else newDf = Df1.union(Df2)

May happened that Df1 is empty and the output is simply: []. In that case I do not need union.

I have it like this but getting error when creating dataframe

if len(Df1) == 0:
  Df2
else:
  newDf=Df1.union(Df2)

-werners- — Thu, 04 Nov 2021 11:50:09 GMT

a dataframe has no 'len' method. use df.count instead.

That being said: it might be a good idea to always assign the result of your if-else to a dataframe. Makes it easier to use:

if df1.count == 0:
  newdf = df2
else:
  newdf = df1.union(df2)

Like that you know the result will always be newdf, instead of df2 or newdf.

Hubert-Dudek — Thu, 04 Nov 2021 12:12:13 GMT

Probably it could be achieved also in pure SPARK SQL using

if(expr1, expr2, expr3)

so expr1 we check is there rows, expr2 we return union, expr3 we return . Not sure it will work, I can check it later.

cconnell — Thu, 04 Nov 2021 13:12:20 GMT

Also try df.head(1).isEmpty

Braxx — Thu, 04 Nov 2021 16:46:07 GMT

You were right. Thanks!