Conditionally create a dataframe

[Braxx]

I would like to implement a simple logic:

if Df1 is empty return Df2 else newDf = Df1.union(Df2)

May happened that Df1 is empty and the output is simply: []. In that case I do not need union.

I have it like this but getting error when creating dataframe

if len(Df1) == 0:
  Df2
else:
  newDf=Df1.union(Df2)

[-werners- · Accepted solution]

a dataframe has no 'len' method. use df.count instead.

That being said: it might be a good idea to always assign the result of your if-else to a dataframe. Makes it easier to use:

if df1.count == 0:
  newdf = df2
else:
  newdf = df1.union(df2)

Like that you know the result will always be newdf, instead of df2 or newdf.

[Braxx]

You were right. Thanks!

[Hubert-Dudek]

Probably it could be achieved also in pure SPARK SQL using

if(expr1, expr2, expr3)

so expr1 we check is there rows, expr2 we return union, expr3 we return . Not sure it will work, I can check it later.

[cconnell]

Also try df.head(1).isEmpty