How are you combining forecasting models for time-series work?

[ThiamLee]

Most time-series forecasting jobs do not come down to one model. Teams run N-BEATS, NHITS, LSTM, and GRU, then struggle to combine them well.

I want to open a discussion on model synthesis. Not data cleaning. Not generative AI. I mean merging several trained forecasters into one stronger prediction.

A few questions for the community:

• How do you weight models when their errors shift across horizons?
• Do you retrain the ensemble, or blend outputs after the fact?
• Which metrics tell you a synthesis beats the strongest single model?

We think about this problem a lot at Remix Labs. We build a time-series synthesis platform for exactly this gap. It layers N-BEATS, NHITS, LSTM, and GRU, then learns how to merge their outputs. The goal is steadier accuracy as conditions change.

I am curious how Databricks users handle this today. Do you lean on MLflow to track competing models? Do you ensemble inside your pipelines, or keep models separate?

Keen to hear what has worked for your teams.

[Islam_hoti]

Hi,

Good topic. A few things that have held up for us.

On weighting across horizons, fit weights per horizon step rather than one global set, and derive them from rolling origin backtests so the errors are genuinely out of sample. Models tend to trade places as the horizon extends, and a single weight vector averages that structure away.

On retrain versus blend after the fact, post hoc blending has been more practical for us. Stacking with a meta learner works, but only if it is trained on out of fold predictions, otherwise you are just fitting leakage. Non negative least squares or ridge as the combiner has consistently beaten anything more elaborate we tried.

The uncomfortable part is your third question. The baseline to beat is not the strongest single model, it is the equal weight average of your models. That is a much harder target than people expect, and a lot of learned combiners fail to clear it out of sample. When we compare, we look at per horizon skill scores rather than one aggregate number, use MASE when pooling across series with different scales, and check whether the improvement is in the mean error or in the tail. Often ensembling delivers stability rather than accuracy, which is valuable but a different claim. A Diebold Mariano test is worth running before declaring a winner.

On the Databricks side, MLflow handles tracking fine, but the useful pattern is logging the per horizon backtest matrix as an artifact rather than only scalar metrics, then wrapping the ensemble as a single pyfunc model so serving does not need to know how many components sit behind it.

Curious whether your platform's learned merge beats an equal weight average across regime shifts, and by how much. That is the number I would want to see.