topic Can't parallelize model training with sc.parallelize, even tough I can run the same code without parallelizing in Data Engineering

Can't parallelize model training with sc.parallelize, even tough I can run the same code without parallelizing

rubenteixeira — Mon, 09 Jan 2023 15:16:03 GMT

I'm training a NeuralProphet for a time series forecasting problem. I'm trying to parallelize my training, but this error is appearing

The folder lightning_logs has a hparams.yaml but it's empty. Is this related to permissions on the cluster? Thanks in advance

Re: Can't parallelize model training with sc.parallelize, even tough I can run the same code without parallelizing

rubenteixeira — Tue, 10 Jan 2023 10:44:50 GMT

How would I check it? The error is coming from the line 21, "train = m.fit(df=df_train, freq="M")", that previously ran with no problems. Here's the code:

When I run the neural_prophet_cv function without the parallelize, this is the result:

I suspect I don't have permissions writing permissions from the cluster

Re: Can't parallelize model training with sc.parallelize, even tough I can run the same code without parallelizing

Debayan — Mon, 09 Jan 2023 22:07:40 GMT

Hi,

Please let us know if this was checked already: