topic Re: Python running far slower than locally, even with large cluster and multiple workers in Machine Learning https://community.databricks.com/t5/machine-learning/python-running-far-slower-than-locally-even-with-large-cluster/m-p/99604#M3801 <P>Just noticed it says Python, in title - it should say Spark.</P> Thu, 21 Nov 2024 12:23:54 GMT JoeAckerman 2024-11-21T12:23:54Z Python running far slower than locally, even with large cluster and multiple workers https://community.databricks.com/t5/machine-learning/python-running-far-slower-than-locally-even-with-large-cluster/m-p/99603#M3800 <P>I have a notebook that is running extremely slowly even when I try to do pretty basic python functions. It is running far slower than locally no matter what I try, this is in spite of using a 32gb 4 core cluster with 4-8 workers. For context, my data is around 7m rows with about 20 columns, so not huge! Any help would be much appreciated.</P> Thu, 21 Nov 2024 12:21:47 GMT https://community.databricks.com/t5/machine-learning/python-running-far-slower-than-locally-even-with-large-cluster/m-p/99603#M3800 JoeAckerman 2024-11-21T12:21:47Z Re: Python running far slower than locally, even with large cluster and multiple workers https://community.databricks.com/t5/machine-learning/python-running-far-slower-than-locally-even-with-large-cluster/m-p/99604#M3801 <P>Just noticed it says Python, in title - it should say Spark.</P> Thu, 21 Nov 2024 12:23:54 GMT https://community.databricks.com/t5/machine-learning/python-running-far-slower-than-locally-even-with-large-cluster/m-p/99604#M3801 JoeAckerman 2024-11-21T12:23:54Z Re: Python running far slower than locally, even with large cluster and multiple workers https://community.databricks.com/t5/machine-learning/python-running-far-slower-than-locally-even-with-large-cluster/m-p/100155#M3806 <P>Please share more information, for example:</P> <UL> <LI>Type of data source</LI> <LI>Type of operations being executed (sharing code if possible)</LI> <LI>Timings of local runs and Databricks runs</LI> </UL> Tue, 26 Nov 2024 19:33:20 GMT https://community.databricks.com/t5/machine-learning/python-running-far-slower-than-locally-even-with-large-cluster/m-p/100155#M3806 cgrant 2024-11-26T19:33:20Z