Hey,
We got two models A and B
Model A is fed from raw data that is firstly Clean / enriched and forecasted
The results from model A are what are fed into model B
the processes for cleaning, enriching, forecasting, model A and model B are all under version control in git
We're looking for a way to automate model history and display to an end user why the result would look different if the same raw data was fed in today Vs the last time it was run
I don't think there is any in built functionality within Databricks to do this... at a high level I was thinking of doing something like the flow in the attached image suggests and was wondering if there was a better solution or is I've fallen into any clangers?
thanks for your help!