topic how to dynamically perform aggregation on all columns in a data frame even when some columns have different types like int , double string datetime or float in pyspark (i have 140-200 columns and need to perform aggregation/avg on each column) in Data Engineering https://community.databricks.com/t5/data-engineering/how-to-dynamically-perform-aggregation-on-all-columns-in-a-data/m-p/8465#M4109 <P>need to aggregate all the numerical columns but need to this dynamically </P> Wed, 01 Mar 2023 15:02:44 GMT STummala 2023-03-01T15:02:44Z how to dynamically perform aggregation on all columns in a data frame even when some columns have different types like int , double string datetime or float in pyspark (i have 140-200 columns and need to perform aggregation/avg on each column) https://community.databricks.com/t5/data-engineering/how-to-dynamically-perform-aggregation-on-all-columns-in-a-data/m-p/8465#M4109 <P>need to aggregate all the numerical columns but need to this dynamically </P> Wed, 01 Mar 2023 15:02:44 GMT https://community.databricks.com/t5/data-engineering/how-to-dynamically-perform-aggregation-on-all-columns-in-a-data/m-p/8465#M4109 STummala 2023-03-01T15:02:44Z Re: how to dynamically perform aggregation on all columns in a data frame even when some columns have different types like int , double string datetime or float in pyspark (i have 140-200 columns and need to perform aggregation/avg on each column) https://community.databricks.com/t5/data-engineering/how-to-dynamically-perform-aggregation-on-all-columns-in-a-data/m-p/8467#M4111 <P>Hi ​@sandeep tummala​&nbsp;, </P><P></P><P>Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.</P><P></P><P>Please help us select the best solution by clicking on "Select As Best" if it does.</P><P></P><P>Your feedback will help us ensure that we are providing the best possible service to you. </P><P></P><P>Thank you!</P><P></P><P></P> Mon, 06 Mar 2023 07:16:13 GMT https://community.databricks.com/t5/data-engineering/how-to-dynamically-perform-aggregation-on-all-columns-in-a-data/m-p/8467#M4111 Anonymous 2023-03-06T07:16:13Z Re: how to dynamically perform aggregation on all columns in a data frame even when some columns have different types like int , double string datetime or float in pyspark (i have 140-200 columns and need to perform aggregation/avg on each column) https://community.databricks.com/t5/data-engineering/how-to-dynamically-perform-aggregation-on-all-columns-in-a-data/m-p/8466#M4110 <P>Hi, Have you tried using the aggregate function which may help in this case? </P><P><A href="https://docs.databricks.com/sql/language-manual/functions/aggregate.html" alt="https://docs.databricks.com/sql/language-manual/functions/aggregate.html" target="_blank">https://docs.databricks.com/sql/language-manual/functions/aggregate.html</A></P> Mon, 06 Mar 2023 04:57:08 GMT https://community.databricks.com/t5/data-engineering/how-to-dynamically-perform-aggregation-on-all-columns-in-a-data/m-p/8466#M4110 Debayan 2023-03-06T04:57:08Z