https://community.databricks.com/t5/generative-ai/large-datasets-in-databricks/m-p/100516#M651 <P>How can I efficiently handle large datasets in Databricks when performing group-by operations to avoid out-of-memory errors? Are there any best practices or optimizations for improving performance, such as partitioning or caching, especially when working with Spark DataFrames?</P> Sun, 01 Dec 2024 05:30:40 GMT maltasa 2024-12-01T05:30:40Z https://community.databricks.com/t5/generative-ai/large-datasets-in-databricks/m-p/100516#M651 <P>How can I efficiently handle large datasets in Databricks when performing group-by operations to avoid out-of-memory errors? Are there any best practices or optimizations for improving performance, such as partitioning or caching, especially when working with Spark DataFrames?</P> Sun, 01 Dec 2024 05:30:40 GMT https://community.databricks.com/t5/generative-ai/large-datasets-in-databricks/m-p/100516#M651 maltasa 2024-12-01T05:30:40Z https://community.databricks.com/t5/generative-ai/large-datasets-in-databricks/m-p/100633#M652 <P>Hi,&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/134438">@maltasa</a>&nbsp;</P><P>I believe this article might help answer your question.</P><P><A href="https://www.databricks.com/discover/pages/optimize-data-workloads-guide" target="_self">Comprehensive Guide to Optimize Databricks, Spark and Delta Lake Workloads</A>&nbsp;</P> Mon, 02 Dec 2024 14:01:42 GMT https://community.databricks.com/t5/generative-ai/large-datasets-in-databricks/m-p/100633#M652 Takuya-Omi 2024-12-02T14:01:42Z