topic Re: Which is quicker: grouping a table that is a join of several others or querying data? in Data Engineering

Which is quicker: grouping a table that is a join of several others or querying data?

markdias — Thu, 06 Oct 2022 14:52:25 GMT

This may be a tricky question, so please bear with me

In a real life scenario, i have a dataframe (i'm using pyspark) called age, with is a groupBy of other 4 dataframes. I join these 4 so at the end I have a few million rows, but after the groupBy the numbers are reduced for some 200 rows.

I then save this dataframe to an s3 bucket.

The question now is:

what is quicker: performing more groupBy in this dataframe, or querying the data i just saved in s3 and then applying the groupBy to it?

The final goal is to save this second groupBy in s3 too.

Re: Which is quicker: grouping a table that is a join of several others or querying data?

Hubert-Dudek — Fri, 14 Oct 2022 12:12:42 GMT

'with is a groupBy of other 4 dataframes' I don't understand it, you can share code.

Faster will be to process everything in one goal usually.

Re: Which is quicker: grouping a table that is a join of several others or querying data?

Anonymous — Tue, 15 Nov 2022 08:48:59 GMT

Hi @Marcos Dias

Hope all is well!

Does @Hubert Dudek (Customer) response were able to resolve your issue, and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: Which is quicker: grouping a table that is a join of several others or querying data?

NhatHoang — Tue, 15 Nov 2022 09:14:38 GMT

Hi @Marcos Dias ,

Frankly, I think we need more detail to answer your question:

Are these 4 dataframes updated their data?
How often you use the groupBy-dataframe?