https://community.databricks.com/t5/data-engineering/pre-partitioning-a-delta-table-to-reduce-suffling-of-wide/m-p/82915#M36773 <P>Hi Maatari!</P><P>How are you doing today?</P><P>When you group data by a column in a Delta table, Spark typically has to shuffle the data to get all the same values together. But if your Delta table is already partitioned by that same column, the shuffling is much less because the data is already nicely organized.</P><P>For example, if your Delta table is partitioned by store_id, and you want to group by store_id&nbsp;to see total sales per store, Spark can do that faster since it doesn't need to move data around as much.</P><P>Also, when you load data from a Delta table into a DataFrame, Spark usually respects the table’s partitioning. So if your table is partitioned by store_id, your DataFrame might also be partitioned that way, which again helps reduce shuffling during operations like <STRONG>groupby.</STRONG></P><P>In short, if you partition your Delta table by the column you plan to group by, it can make your queries run a lot smoother!&nbsp;</P><P>Have a good day.</P> Wed, 14 Aug 2024 03:41:30 GMT Brahmareddy 2024-08-14T03:41:30Z https://community.databricks.com/t5/data-engineering/pre-partitioning-a-delta-table-to-reduce-suffling-of-wide/m-p/82874#M36758 <P>Assuming i need to perfom a groupby i.e. aggregation on a dataset stored in a delta table. If the delta table is partitioned by the field by which to group, can that have an impact on the suffling that the groupby would normally cause ?&nbsp;</P><P>As a connected question, one can ask is there any correlation between how a delta table is partitioned and how the data is put into the dataframe partition when loading the data ?</P> Tue, 13 Aug 2024 13:02:25 GMT https://community.databricks.com/t5/data-engineering/pre-partitioning-a-delta-table-to-reduce-suffling-of-wide/m-p/82874#M36758 Maatari 2024-08-13T13:02:25Z https://community.databricks.com/t5/data-engineering/pre-partitioning-a-delta-table-to-reduce-suffling-of-wide/m-p/82932#M36783 <P>Hi&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/102834">@Maatari</a>, Thanks for reaching out! Please review the responses and let us know which best addresses your question. Your feedback is valuable to us and the community.</P> <P>&nbsp;</P> <P>If the response resolves your issue, kindly mark it as the accepted solution. This will help close the thread and assist others with similar queries.</P> <P>&nbsp;</P> <P>We appreciate your participation and are here if you need further assistance!</P> Wed, 14 Aug 2024 08:07:13 GMT https://community.databricks.com/t5/data-engineering/pre-partitioning-a-delta-table-to-reduce-suffling-of-wide/m-p/82932#M36783 Retired_mod 2024-08-14T08:07:13Z