@Kevin Kimโ : Hope the below answer gives you some pointers to think, test and try, implement.
The choice of numerical solver may affect the speed and accuracy of the optimization process when fitting the SARIMAX model. The default solver, lbfgs, is a gradient-based optimization method that is known to work well for small datasets with relatively few parameters to estimate. However, for larger datasets or models with more parameters, it may be necessary to use a different solver, such as Newton-Raphson or BFGS, to achieve better convergence and faster performance.
Regarding network latency, this refers to the time it takes for data to travel between your Databricks cluster and any other external data sources or services that your code may be accessing. If your code is making frequent or large data requests to an external source, network latency can become a significant bottleneck that affects the overall performance of your computations. To mitigate this, you may want to consider optimizing your code to reduce the number of data requests or to preprocess and cache data locally in your Databricks cluster. Additionally, you may want to consider using a higher-performing network or optimizing your network configuration to reduce latency.
