Can't parallelize model training with sc.parallelize, even tough I can run the same code without parallelizing

rubenteixeira
Databricks Partner

I'm training a NeuralProphet for a time series forecasting problem. I'm trying to parallelize my training, but this error is appearing

imageimage.pngThe folder lightning_logs has a hparams.yaml but it's empty. Is this related to permissions on the cluster? Thanks in advance