cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

auto.arima in R keeps running without end

arkobose
New Contributor III

I have a time-series that I am trying to model using auto.arima in R. Upon execution on a cluster runtime with 96GB memory and 48 cores, I observed that the code kept running without any end in sight. It ran for over a day without finishing.

Curious, I ran the same code on the same time-series on my modest laptop (32GB RAM, 10 cores), the code finished execution in under 5 minutes.

I am at a complete loss to understand what is taking the code I am running on my Databricks cluster runtime take so long!

For completeness, the code I am trying to execute is simply this:

baselines_daily <- auto.arima(daily_network_efficiency,
                                xreg = F,
                                seasonal = FALSE,
                                stepwise = FALSE,
                                parallel = TRUE,
                                approximation = FALSE)
 
where daily_network_efficiency is my time-series.
Any help in shedding light on this mystery will be greatly appreciated.
 
Thank you!

 

1 ACCEPTED SOLUTION

Accepted Solutions

arkobose
New Contributor III

Okay, this is resolved. Windows doesn't allow forking, so parallel = TRUE was having no effect. But Databricks runtime runs on Linux, which enables forking. So keeping parallel = TRUE seems to be having the opposite effect than the anticipated one: due to process overheads, and the fact that parallel only works when stepwise = FALSE, thereby forcing the model to perform a brute-force search.

Tl;dr: I turned off parallel and the execution finished fairly quickly.

View solution in original post

1 REPLY 1

arkobose
New Contributor III

Okay, this is resolved. Windows doesn't allow forking, so parallel = TRUE was having no effect. But Databricks runtime runs on Linux, which enables forking. So keeping parallel = TRUE seems to be having the opposite effect than the anticipated one: due to process overheads, and the fact that parallel only works when stepwise = FALSE, thereby forcing the model to perform a brute-force search.

Tl;dr: I turned off parallel and the execution finished fairly quickly.