topic How to pick the right cluster for your workflow in Data Engineering https://community.databricks.com/t5/data-engineering/how-to-pick-the-right-cluster-for-your-workflow/m-p/75935#M35106 <P>Hi All,</P><P>I am attempting to execute a workflow on various job clusters, including general-purpose and memory-optimized clusters. My main bottleneck is that data is being written to disk because I’m running out of RAM. This is due to the large dataset that I need to load into memory.</P><P>The size of the dataset is unavoidable, but the computations are straightforward.</P><P>How should I go about selecting the appropriate cluster? Is there a useful guide for choosing the right cluster that I could follow going forward?</P><P>Thanks,</P><P>Aidzillafont</P> Thu, 27 Jun 2024 11:25:12 GMT Aidzillafont 2024-06-27T11:25:12Z How to pick the right cluster for your workflow https://community.databricks.com/t5/data-engineering/how-to-pick-the-right-cluster-for-your-workflow/m-p/75935#M35106 <P>Hi All,</P><P>I am attempting to execute a workflow on various job clusters, including general-purpose and memory-optimized clusters. My main bottleneck is that data is being written to disk because I’m running out of RAM. This is due to the large dataset that I need to load into memory.</P><P>The size of the dataset is unavoidable, but the computations are straightforward.</P><P>How should I go about selecting the appropriate cluster? Is there a useful guide for choosing the right cluster that I could follow going forward?</P><P>Thanks,</P><P>Aidzillafont</P> Thu, 27 Jun 2024 11:25:12 GMT https://community.databricks.com/t5/data-engineering/how-to-pick-the-right-cluster-for-your-workflow/m-p/75935#M35106 Aidzillafont 2024-06-27T11:25:12Z Re: How to pick the right cluster for your workflow https://community.databricks.com/t5/data-engineering/how-to-pick-the-right-cluster-for-your-workflow/m-p/75943#M35110 <P>Hello&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/87158">@Aidzillafont</a>&nbsp;,</P> <P>Greetings!</P> <P>Please find below the document which explains the Compute configuration best practices</P> <P>Doc:&nbsp;<A href="https://docs.databricks.com/en/compute/cluster-config-best-practices.html" target="_blank">https://docs.databricks.com/en/compute/cluster-config-best-practices.html</A></P> <P>I hope this helps you!</P> <P>Regards,</P> <P>Ravi</P> Thu, 27 Jun 2024 12:49:35 GMT https://community.databricks.com/t5/data-engineering/how-to-pick-the-right-cluster-for-your-workflow/m-p/75943#M35110 Ravivarma 2024-06-27T12:49:35Z