cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

slow running query

joakon
New Contributor III

Hi All,

I would you to get some ideas on how to improve performance on a data frame with around 10M rows.

adls- gen2

df1 =source1 , format , parquet ( 10 m)

df2 =source2 , format , parquet ( 10 m)

df = join df1 and df2 type =inner join

df.count() is taking for ever.

trying to join the above sources and aggregate them and write back to adls .

1 ACCEPTED SOLUTION

Accepted Solutions

LandanG
Honored Contributor
Honored Contributor

@raghu maremanda​ It's hard to provide an answer without having more info. Can you add the actual code used in the join, as well as the total data size, & cluster configuration (note types & number of nodes)

View solution in original post

5 REPLIES 5

LandanG
Honored Contributor
Honored Contributor

@raghu maremanda​ It's hard to provide an answer without having more info. Can you add the actual code used in the join, as well as the total data size, & cluster configuration (note types & number of nodes)

labtech
Valued Contributor II

Which size of parquet file, if is too small you can try with pandas then compare to pyspark

sher
Valued Contributor II

you can use ShuffleHashJoin to improve

Aviral-Bhardwaj
Esteemed Contributor III

yeah this is easy you can do some performance tuning in your cluster and it will work, you can use auto broadcast join configuration or other where you can set up your performance tuning

Aviral-Bhardwaj
Esteemed Contributor III

hey @raghu maremanda​ did you get any answer if yes ,please update here, by that other people can also get the solution

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.