05-01-2023 12:21 AM
I have a workflow with a model which trains every sunday in Azure Databricks. Sometimes the workflow fails as the max wait time is exceeded (currently I am using 1200 seconds). To solve the problem I was thinking of either increasing the wait time or increasing the size of the cluster used.
This made me wonder, which is better (and cheaper) of the two options below:
Or is there a third and better solution?
05-03-2023 05:04 AM
You´ll need to play with those two options, but, since initial iterations of training a machine learning model are often experimental, a smaller cluster is a good choice. A smaller cluster will also reduce the impact of shuffles. Recommended worker types are storage optimized with Delta Caching enabled to account for repeated reads of the same data and to enable caching of training data. If the compute and storage options provided by storage-optimized nodes are not sufficient, consider GPU-optimized nodes. A possible downside is the lack of Delta Caching support with these nodes.
Also, if you are talking about training deep learning models, check best practices, and if you are using PyTorch, the new. TorchDistributor.
https://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-model/dl-best-practices
PS; Check #DAIS2023 talks, the creator of PyTorch is giving a keynote
05-02-2023 06:44 PM
Does increasing or decreasing size of cluster helps for this process ?
05-03-2023 05:04 AM
You´ll need to play with those two options, but, since initial iterations of training a machine learning model are often experimental, a smaller cluster is a good choice. A smaller cluster will also reduce the impact of shuffles. Recommended worker types are storage optimized with Delta Caching enabled to account for repeated reads of the same data and to enable caching of training data. If the compute and storage options provided by storage-optimized nodes are not sufficient, consider GPU-optimized nodes. A possible downside is the lack of Delta Caching support with these nodes.
Also, if you are talking about training deep learning models, check best practices, and if you are using PyTorch, the new. TorchDistributor.
https://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-model/dl-best-practices
PS; Check #DAIS2023 talks, the creator of PyTorch is giving a keynote
05-18-2023 02:30 AM
Hi @Andreas Kaae
Thank you for posting your question in our community! We are happy to assist you.
To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?
This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group