Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
The same Scoring notebook is run 3 times as 3 Jobs. The jobs are identical, same PetaStorm code, CPU cluster config ( not Spot cluster) and data but have varying elapsed runtimes. Elapsed runtimes - Job 1: 3 hours, Job 2 39 min, Job 3 42 min. What would cause the large Job 1 variation?
The same Scoring code is ported to run on GPU cluster. Updated environment and libraries were required. Same data. Runtimes: JobG 1 GPU single node cluster 67 min. JobG 2 multi-node 1-4 cluster 73 min. The JobG 2 multi-node did not run faster than JobG 1 single node. Looking a logs, the nodes run serially and not parallel. node 1 runs and stops, node 2 runs and stops, node 3 runs and stops, node 4 runs and stops. Is there a configuration parameter that is required for parallel node processing? Or another factor ?
Could you please confirm, if you have set any parameters for the best model? Is this stop after running some epochs if there is no improvement in the model performance?
Join Us as a Local Community Builder!
Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!
