topic Re: Cluster termination issue (It was working fine with same code previously) in Machine Learning

Cluster termination issue (It was working fine with same code previously)

Humi1245 — Tue, 26 Sep 2023 07:48:00 GMT

Hi, I am running arima code on databricks commmunity edition. Last time, it worked but now the cluster gets terminated at this command and says its completed. but i cant further plot the predictions since the cluster is terminated.

# Fit the AutoARIMA model

model = auto_arima(train['Price'], seasonal=True, m=365, stepwise=True, suppress_warnings=True, error_action='ignore', trace=False)

What is the issue?

Re: Cluster termination issue (It was working fine with same code previously)

Kumaran — Tue, 03 Oct 2023 14:47:20 GMT

Hi @Humi1245

Thank you for your question in the Databricks community

One possible issue may be that the cluster is running out of memory while trying to fit the AutoARIMA model. You can try increasing the size of the cluster and see if that resolves the issue.

You can check the status of the cluster, the resource utilization in the cluster and the network connectivity to the cluster.

Re: Cluster termination issue (It was working fine with same code previously)

Annapurna_Hiriy — Wed, 11 Oct 2023 07:33:17 GMT

@Humi1245 You should check the cluster logs to see the reason for cluster termination. Also, check the metrics to understand the memory/CPU consumption.

Re: Cluster termination issue (It was working fine with same code previously)

Tharun-Kumar — Wed, 11 Oct 2023 09:56:06 GMT

@Humi1245

You can view the event log of the cluster to understand the cause of the cluster termination. If it is due to gc issues, then we have to understand the reason behind high driver memory utilization which caused the GC.

If it's due to inactivity, then the cluster has been idle for long time.