cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Serhii
by • Contributor
  • 1678 Views
  • 2 replies
  • 0 kudos

Resolved! Horovod Databricks Job - custom module not found error

We have used the following example to successfully create a distributed deep learning training notebook https://www.databricks.com/blog/2022/09/07/accelerating-your-deep-learning-pytorch-lightning-databricks.html that works as expected.We now want to...

Data Engineering
Reply
  • 1678 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Sergii Ivakhno​ ​, We haven’t heard from you since the last response from me​ ​, and I was checking back to see if my suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to others.Als...

  • 0 kudos
1 More Replies
User16857281869
by • New Contributor II
  • 1019 Views
  • 1 replies
  • 0 kudos

How do I benefit from parallelisation when doing machine learning?

There are in principle four distinct ways of using parallelisation when doing machine learning. Any combination of these can speed up the whole pipeline significantly.1) Using spark distributed processing in feature engineering 2) When the data set...

Data Engineering
Reply
  • 1019 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
sean_owen
Esteemed Contributor III
  • 0 kudos

Good summary! yes those are the main strategies I can think of.

  • 0 kudos
User16788317466
by • Esteemed Contributor III
  • 670 Views
  • 1 replies
  • 0 kudos

When can Horovod be used for an ML problem?

When can Horovod be used for an ML problem?

Data Engineering
Reply
  • 670 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16788317466
Esteemed Contributor III
  • 0 kudos

Only when you have a gradient-descent problem. Pytorch and Tensorflow are the only candidate frameworks to use here. When using Horovod, start with single node, multi-GPU and measure training performance. If this is not sufficient, look at a multi-no...

  • 0 kudos
Labels