topic GroupBy in a multi node environment in Data Engineering https://community.databricks.com/t5/data-engineering/groupby-in-a-multi-node-environment/m-p/16968#M11053 <P></P> <P>I have a group of rows with Information on a nested product calls. </P><P></P>example- <P>Trxn1-product1-caller1-local1</P> <P>Trxn1-Product1-local1-local2</P> <P>Trxn1-Product1-local2-local3</P> <P>here’s is a expected calls for a product </P> <P>product1-caller1-local1</P> <P>Product1-local1-local2</P> <P>Product1-local2-local3</P> <P>Product1-local3-local4</P> <P>Product1-local4-local5</P> <P></P><P></P>now I need to create one additional record to my dataframe <P>Trxn1-product1-local3-local4. Picking only this coz, that’s the first missing link in the chain based on where it failed. </P><P></P> <P>Like Trxn1, I have 10000s of them and I need to group these by transaction Id and apply the condition based on the product. </P><P></P> <P>Trying if for loop is taking a </P> <P></P> Sat, 31 Jul 2021 21:48:02 GMT RajuNagarajan 2021-07-31T21:48:02Z GroupBy in a multi node environment https://community.databricks.com/t5/data-engineering/groupby-in-a-multi-node-environment/m-p/16968#M11053 <P></P> <P>I have a group of rows with Information on a nested product calls. </P><P></P>example- <P>Trxn1-product1-caller1-local1</P> <P>Trxn1-Product1-local1-local2</P> <P>Trxn1-Product1-local2-local3</P> <P>here’s is a expected calls for a product </P> <P>product1-caller1-local1</P> <P>Product1-local1-local2</P> <P>Product1-local2-local3</P> <P>Product1-local3-local4</P> <P>Product1-local4-local5</P> <P></P><P></P>now I need to create one additional record to my dataframe <P>Trxn1-product1-local3-local4. Picking only this coz, that’s the first missing link in the chain based on where it failed. </P><P></P> <P>Like Trxn1, I have 10000s of them and I need to group these by transaction Id and apply the condition based on the product. </P><P></P> <P>Trying if for loop is taking a </P> <P></P> Sat, 31 Jul 2021 21:48:02 GMT https://community.databricks.com/t5/data-engineering/groupby-in-a-multi-node-environment/m-p/16968#M11053 RajuNagarajan 2021-07-31T21:48:02Z