https://community.databricks.com/t5/data-engineering/cpu-and-gpu-elapse-runtimes/m-p/39675#M27053 <P>I have to 2 questions about elapsed job runtimes.&nbsp;</P><OL><LI>The same Scoring notebook is run 3 times as 3 Jobs.&nbsp;&nbsp; The jobs are identical, same PetaStorm code, CPU cluster config ( not Spot cluster) and data but have varying elapsed runtimes.&nbsp;&nbsp; Elapsed runtimes - Job 1: &nbsp;3 hours,&nbsp; Job 2 39 min, Job 3 &nbsp;42 min.&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;What would cause the large Job 1 variation?</LI></OL><P>&nbsp;</P><OL><LI>The same Scoring code is ported to run on GPU cluster.&nbsp; Updated environment and libraries were required.&nbsp;&nbsp; Same data.&nbsp; Runtimes: JobG 1 GPU single node cluster 67 min.&nbsp; JobG 2 &nbsp;multi-node 1-4 cluster &nbsp;73 min.&nbsp;&nbsp; The JobG 2&nbsp; multi-node did not run faster than JobG 1 single node.&nbsp;&nbsp; Looking a logs, the nodes run serially and not parallel.&nbsp; node 1 runs and stops, node 2 runs and stops, node 3 runs and stops, node 4 runs and stops.&nbsp;&nbsp; Is there a configuration parameter that is required for parallel node processing?&nbsp; Or another factor ?</LI></OL><P>&nbsp;</P><P>Thanks for help.</P> Fri, 11 Aug 2023 17:03:37 GMT NathanLaw 2023-08-11T17:03:37Z https://community.databricks.com/t5/data-engineering/cpu-and-gpu-elapse-runtimes/m-p/39675#M27053 <P>I have to 2 questions about elapsed job runtimes.&nbsp;</P><OL><LI>The same Scoring notebook is run 3 times as 3 Jobs.&nbsp;&nbsp; The jobs are identical, same PetaStorm code, CPU cluster config ( not Spot cluster) and data but have varying elapsed runtimes.&nbsp;&nbsp; Elapsed runtimes - Job 1: &nbsp;3 hours,&nbsp; Job 2 39 min, Job 3 &nbsp;42 min.&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;What would cause the large Job 1 variation?</LI></OL><P>&nbsp;</P><OL><LI>The same Scoring code is ported to run on GPU cluster.&nbsp; Updated environment and libraries were required.&nbsp;&nbsp; Same data.&nbsp; Runtimes: JobG 1 GPU single node cluster 67 min.&nbsp; JobG 2 &nbsp;multi-node 1-4 cluster &nbsp;73 min.&nbsp;&nbsp; The JobG 2&nbsp; multi-node did not run faster than JobG 1 single node.&nbsp;&nbsp; Looking a logs, the nodes run serially and not parallel.&nbsp; node 1 runs and stops, node 2 runs and stops, node 3 runs and stops, node 4 runs and stops.&nbsp;&nbsp; Is there a configuration parameter that is required for parallel node processing?&nbsp; Or another factor ?</LI></OL><P>&nbsp;</P><P>Thanks for help.</P> Fri, 11 Aug 2023 17:03:37 GMT https://community.databricks.com/t5/data-engineering/cpu-and-gpu-elapse-runtimes/m-p/39675#M27053 NathanLaw 2023-08-11T17:03:37Z https://community.databricks.com/t5/data-engineering/cpu-and-gpu-elapse-runtimes/m-p/40128#M27134 <P>Hi&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/62613">@NathanLaw</a>,&nbsp;</P><P>Could you please confirm, if you have set any parameters for the best model? Is this stop after running some epochs if there is no improvement in the model performance?&nbsp;</P> Thu, 17 Aug 2023 00:33:28 GMT https://community.databricks.com/t5/data-engineering/cpu-and-gpu-elapse-runtimes/m-p/40128#M27134 shyam_9 2023-08-17T00:33:28Z