topic Re: slow running query in Data Engineering

slow running query

joakon — Fri, 16 Dec 2022 21:10:40 GMT

Hi All,

I would you to get some ideas on how to improve performance on a data frame with around 10M rows.

adls- gen2

df1 =source1 , format , parquet ( 10 m)

df2 =source2 , format , parquet ( 10 m)

df = join df1 and df2 type =inner join

df.count() is taking for ever.

trying to join the above sources and aggregate them and write back to adls .

Re: slow running query

LandanG — Fri, 16 Dec 2022 21:40:33 GMT

@raghu maremanda It's hard to provide an answer without having more info. Can you add the actual code used in the join, as well as the total data size, & cluster configuration (note types & number of nodes)

Re: slow running query

labtech — Sat, 17 Dec 2022 07:01:16 GMT

Which size of parquet file, if is too small you can try with pandas then compare to pyspark

Re: slow running query

sher — Sat, 17 Dec 2022 07:41:03 GMT

you can use ShuffleHashJoin to improve

Re: slow running query

Aviral-Bhardwaj — Sun, 18 Dec 2022 06:25:07 GMT

yeah this is easy you can do some performance tuning in your cluster and it will work, you can use auto broadcast join configuration or other where you can set up your performance tuning

Re: slow running query

Aviral-Bhardwaj — Sat, 24 Dec 2022 04:37:33 GMT

hey @raghu maremanda did you get any answer if yes ,please update here, by that other people can also get the solution