Kannathasan
Databricks Partner

Could you please try increase the partition the Dataframe by doing repartition() before you merge.