Can't parallelize model training with sc.parallelize, even tough I can run the same code without parallelizing

[rubenteixeira]

I'm training a NeuralProphet for a time series forecasting problem. I'm trying to parallelize my training, but this error is appearing

The folder lightning_logs has a hparams.yaml but it's empty. Is this related to permissions on the cluster? Thanks in advance

[Debayan]

Hi,

Please let us know if this was checked already:

[rubenteixeira]

How would I check it? The error is coming from the line 21, "train = m.fit(df=df_train, freq="M")", that previously ran with no problems. Here's the code:

When I run the neural_prophet_cv function without the parallelize, this is the result:

I suspect I don't have permissions writing permissions from the cluster