@orian hindi​ :

Regarding your questions:

Q1: The error message you are seeing is likely related to a segmentation fault, which can occur due to various reasons such as memory access violations or stack overflows. It could be caused by several factors, including a bug in your code or issues with your cluster's configuration. I recommend checking the resource allocation for your workers, such as the memory, CPU, and GPU resources, to ensure that they are appropriate for your training workload. Also, check if the version compatibility between PyTorch, Horovod, and Petastorm are correct. You may also want to look into using distributed PyTorch's

DistributedDataParallel (DDP) module instead of Horovod for distributed training.

Q2: If you are using PyTorch, you can manually log metrics and parameters to MLflow by using the mlflow.log_metric() and mlflow.log_param() functions. You can call these functions during training and evaluation to log the relevant metrics and parameters. Additionally, you can use MLflow's tracking UI to log and view the results of your training runs. For PyTorch models, you can use the mlflow.pytorch.log_model() function to log your model after training.

Q3: If you installed imgaug on the driver node, it will not automatically be available on the worker nodes. You will need to ensure that the imgaug package is installed on all nodes in your cluster that are running your training code. One way to do this is to include imgaug in your environment setup script or use a container that has imgaug installed.