How do you train forecasting models for events that never happened?

[ThiamLee]

Most of my time-series work hits the same wall. The models only learn from scenarios that already happened.

Rare events barely show up in the training data. Demand spikes, stockouts, and extreme regimes are exactly the cases I care about. They are also the ones history is thin on.

Lately I have been testing synthetic time-series data as a way around this. The idea is to generate new scenarios from patterns already in your data, then train or stress-test against them.

One tool I came across is Remix Labs. It synthesizes new time-series datasets from data you already have. The pipeline is no-code, and it runs models like N-BEATS, NHITS, LSTM, and GRU under the hood. No new data collection needed.

Curious how others handle this. Do you augment with synthetic data, or stick to real historical records only? What has worked for you?

[ivanvyd]

@ThiamLee thanks for sharing, this is a really interesting problem and useful topic to discuss.

I'd be open to trying augmentation here. I'd compare the same model with and without synthetic data on identical rolling backtests, keeping the test data real and fitting the generator only on each training split. I'd also check real spike periods separately, so the overall score doesn't hide weaker results there.

For events you've never observed, realistic, domain-reviewed scenarios could help with stress testing.Those results are useful, but they still depend on the assumptions behind the scenarios.

One wrinkle with stockouts: sales can understate demand when inventory runsout, so I'd account for that before augmentation.

Have you had a chance to test whether the improvements carry over to held-out real events?

[ivanvyd]

One other thing I'd probably validate is whether the synthetic spikes preserve the relationships with things like promotions, price, holidays, and inventory. A spike can look realistic on its own but still teach the model the wrong context if those relationships get distorted.

[niteshm · Accepted solution]

Hey @ThiamLee , great question, and one I've wrestled with too. Haven't tried Remix Labs specifically, tho.

My take, after going back and forth between pure historical data and augmentation: real data first, augment second, but only where augmentation earns its place.

Three things that helped, in order:

1. Fix censored demand before generating anything.
   When you're out of stock, your sales numbers stop measuring demand. They just measure what was on the shelf. So your data quietly under-reports demand exactly when it spiked. I learned this the hard way. Flag those periods and fill in your best guess of true demand. A simple rolling average of nearby non-stockout days works fine. On one of my datasets this one step beat every augmentation trick I tried.
2. Forecast the range, not a single number.
   For rare events, one predicted number is nearly useless. What you need is the range of what could happen. Train your model to predict quantiles (P10/P50/P90) instead of a single value. If you want intervals that hold up under regime shifts, add conformalized intervals calibrated on your latest backtest fold.
3. Augment like a skeptic.
   When you do generate synthetic data, start simple.The fancy generators like TimeGAN make data that looks real, but they smooth out the extremes, which is the very thing you are after. Keep them for scenario stress-testing. Three rules I never break: fit the generator inside each training fold only, never on the whole dataset. Judge the result on real held-out spikes, never on realism scores. And check the synthetic data still respects your real relationships.

Implementation on Databricks:

1. Bronze: raw sales + inventory/stockout flags + covariates (promo calendar, price, holidays, weather) as Delta tables.
2. Silver: de-censored latent-demand table + causal lag/rolling features, computed in Spark with no future leakage.
3. Baselines: StatsForecast seasonal-naive, then MLForecast + LightGBM quantiles trained globally. Rolling-origin backtests, spike-slice metrics reported separately.
4. Augmentation branch: cheap generators first, one MLflow run per (model × fold), with vs. without augmentation. Promote only on real held-out spike wins.
5. Champion model to Unity Catalog registry, conformal intervals on top for regime-shift coverage.
6. Gold: scheduled scoring writes forecasts here, joined with actuals. This is where monitoring lives. When spike-period error or interval coverage drifts, it triggers a retrain.

Give it a shot and report back. I'm curious what works and what breaks on your data. Good luck!!