slow running query

[joakon]

Hi All,

I would you to get some ideas on how to improve performance on a data frame with around 10M rows.

adls- gen2

df1 =source1 , format , parquet ( 10 m)

df2 =source2 , format , parquet ( 10 m)

df = join df1 and df2 type =inner join

df.count() is taking for ever.

trying to join the above sources and aggregate them and write back to adls .

[LandanG · Accepted solution]

@raghu maremanda​ It's hard to provide an answer without having more info. Can you add the actual code used in the join, as well as the total data size, & cluster configuration (note types & number of nodes)

[labtech]

Which size of parquet file, if is too small you can try with pandas then compare to pyspark

[sher]

you can use ShuffleHashJoin to improve

[Aviral-Bhardwaj]

yeah this is easy you can do some performance tuning in your cluster and it will work, you can use auto broadcast join configuration or other where you can set up your performance tuning

[Aviral-Bhardwaj]

hey @raghu maremanda​ did you get any answer if yes ,please update here, by that other people can also get the solution