Horovod Databricks Job - custom module not found error

Serhii — Tue, 01 Nov 2022 17:13:46 GMT

We have used the following example to successfully create a distributed deep learning training notebook https://www.databricks.com/blog/2022/09/07/accelerating-your-deep-learning-pytorch-lightning-databricks.html that works as expected.

We now want to run this notebook as a task in the Job Compute Workflow, which essentially runs the same code but using Databricks jobs. This surprisingly gives us the error:

INFO:HorovodRunner:Start training.
Warning: Permanently added '172.17.131.218' (ECDSA) to the list of known hosts.
Warning: Permanently added '172.17.162.215' (ECDSA) to the list of known hosts.
[1,1]<stderr>:Traceback (most recent call last):
[1,1]<stderr>:  File "<string>", line 1, in <module>
[1,1]<stderr>:ModuleNotFoundError: No module named 'training'

The training here is the small python module file in the same folder which contains re-usable library functions. My guess is that the top level import code in the notebook is executed on worker node which may not have that file. But I am confused why this is happening:

Shouldn't the horovod just work/pass to workers functions already loaded in the environment that are specifically provided in the call to HorovodRunner.run
Why we don't see on interactive cluster that runs the same notebook

Thanks for your help

topic Horovod Databricks Job - custom module not found error in Data Engineering

Horovod Databricks Job - custom module not found error