Training models on big or small clusters

[akc]

I have a workflow with a model which trains every sunday in Azure Databricks. Sometimes the workflow fails as the max wait time is exceeded (currently I am using 1200 seconds). To solve the problem I was thinking of either increasing the wait time or increasing the size of the cluster used.

This made me wonder, which is better (and cheaper) of the two options below:

1. Train the model on a bigger and more expensive cluster which will hopefully reduce the time used
2. Train the model on a smaller and cheaper cluster and then simply increase the wait time

Or is there a third and better solution?

[Priyag1]

Does increasing or decreasing size of cluster helps for this process ?

[luis_herrera · Accepted solution]

You´ll need to play with those two options, but, since initial iterations of training a machine learning model are often experimental, a smaller cluster is a good choice. A smaller cluster will also reduce the impact of shuffles. Recommended worker types are storage optimized with Delta Caching enabled to account for repeated reads of the same data and to enable caching of training data. If the compute and storage options provided by storage-optimized nodes are not sufficient, consider GPU-optimized nodes. A possible downside is the lack of Delta Caching support with these nodes.

Also, if you are talking about training deep learning models, check best practices, and if you are using PyTorch, the new. TorchDistributor.

https://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-model/dl-best-practices

PS; Check #DAIS2023 talks, the creator of PyTorch is giving a keynote

[Anonymous]

Hi @Andreas Kaae​ 

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!