Conditionally create a dataframe

[Braxx]

I would like to implement a simple logic:

if Df1 is empty return Df2 else newDf = Df1.union(Df2)

May happened that Df1 is empty and the output is simply: []. In that case I do not need union.

I have it like this but getting error when creating dataframe

if len(Df1) == 0:
  Df2
else:
  newDf=Df1.union(Df2)

[Kaniz]

Hi @ Braxx! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.

[-werners- · Accepted solution]

a dataframe has no 'len' method. use df.count instead.

That being said: it might be a good idea to always assign the result of your if-else to a dataframe. Makes it easier to use:

if df1.count == 0:
  newdf = df2
else:
  newdf = df1.union(df2)

Like that you know the result will always be newdf, instead of df2 or newdf.

[Braxx]

You were right. Thanks!

[Hubert-Dudek]

Probably it could be achieved also in pure SPARK SQL using

if(expr1, expr2, expr3)

so expr1 we check is there rows, expr2 we return union, expr3 we return . Not sure it will work, I can check it later.

[cconnell]

Also try df.head(1).isEmpty