cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Union Multiple dataframes in loop, with different schema

KKo
Contributor III

With in a loop I have few dataframes created. I can union them with out an issue if they have same schema using (df_unioned = reduce(DataFrame.unionAll, df_list). Now my problem is how to union them if one of the dataframe in df_list has different number of columns? I thought, reduce(df_unioned=DataFrame.unionByName, df_list, allowMissingColumns=True) would solve the issue but it is giving me error: reduce() takes no keyword arguments. Thanks in advance. Let me know if you need any details in the question.

1 ACCEPTED SOLUTION

Accepted Solutions

KKo
Contributor III

@Joseph Kambourakis​  I found a way to achieve this. using the function

def union_all(dfs):

 if len(dfs) > 1:

  return dfs[0].unionByName(union_all(dfs[1:]), allowMissingColumns=True)

 else:

  return dfs[0]

View solution in original post

4 REPLIES 4

Anonymous
Not applicable

Union doesn't work if they have different schemas and columns. If you do need to union dataframes with different schemas, just add columns of nulls for anything missing to get them to the same schema.

KKo
Contributor III

@Joseph Kambourakis​  I found a way to achieve this. using the function

def union_all(dfs):

 if len(dfs) > 1:

  return dfs[0].unionByName(union_all(dfs[1:]), allowMissingColumns=True)

 else:

  return dfs[0]

Kaniz
Community Manager
Community Manager

Awesome!!

@Kris Koirala​ , Thanks for sharing your solution here !!

anoopunni
New Contributor II

Hi,
I have come across same scenario, using reduce() and unionByname we can implement the solution as below:

val lstDF: List[Datframe] = List(df1,df2,df3,df4,df5)

val combinedDF = lstDF.reduce((df1, df2) => df1.unionByName(df2, allowMissingColumns = true))

#Scala # Spark #multiple schema

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.