cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

slow running query

joakon
New Contributor III

Hi All,

I would you to get some ideas on how to improve performance on a data frame with around 10M rows.

adls- gen2

df1 =source1 , format , parquet ( 10 m)

df2 =source2 , format , parquet ( 10 m)

df = join df1 and df2 type =inner join

df.count() is taking for ever.

trying to join the above sources and aggregate them and write back to adls .

1 ACCEPTED SOLUTION

Accepted Solutions

LandanG
Honored Contributor
Honored Contributor

@raghu maremanda​ It's hard to provide an answer without having more info. Can you add the actual code used in the join, as well as the total data size, & cluster configuration (note types & number of nodes)

View solution in original post

5 REPLIES 5

LandanG
Honored Contributor
Honored Contributor

@raghu maremanda​ It's hard to provide an answer without having more info. Can you add the actual code used in the join, as well as the total data size, & cluster configuration (note types & number of nodes)

labtech
Valued Contributor II

Which size of parquet file, if is too small you can try with pandas then compare to pyspark

sher
Valued Contributor II

you can use ShuffleHashJoin to improve

Aviral-Bhardwaj
Esteemed Contributor III

yeah this is easy you can do some performance tuning in your cluster and it will work, you can use auto broadcast join configuration or other where you can set up your performance tuning

Aviral-Bhardwaj
Esteemed Contributor III

hey @raghu maremanda​ did you get any answer if yes ,please update here, by that other people can also get the solution

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!