Banking · Insurance · Credit Risk · Any Team Running More Than One Model — From Copy-Pasted Pipelines to One Shared Pipeline and One YAML per Model
By @SrinuB in collaboration with @sridharplv
Your first model went to production beautifully. Your fifth model is a copy of the first, with three notebooks edited and a job YAML nobody remembers changing.
Every ML platform I have worked on follows the same arc. The first model gets a careful pipeline. The second gets a copy of it. By the fifth, there are five slightly different versions of "the" training notebook, a bug fix that landed in two of them, and a promotion step that works differently depending on who wrote it. That drift is where the 2 AM incidents and the audit findings come from. It is rarely the model itself.
This article is about removing that drift. I'll walk through the blueprint we built for a large financial institution's credit-risk modelling team, where a model is a single config.yml file and everything else is a shared, tested pipeline. The worked example is a PD (probability of default) scorecard, but the pattern applies to any team that runs more than a handful of models.
The problem: every model becomes its own snowflake
Strip away the industry and the pattern looks like this. A model needs the same lifecycle every time: prepare data, engineer features, train, evaluate against what is live, get approved, deploy, score, monitor, retrain. What differs between models is a small set of decisions: which table, which target, which algorithm, which thresholds.
When those decisions are expressed as code in a copied notebook, four things go wrong at once:
- Fixes don't propagate. A bug fixed in model A's evaluation step stays in models B to E until someone notices.
- Promotion logic diverges. One model auto-promotes, another needs a manual alias flip, a third has a check nobody can find.
- Reviews get expensive. A reviewer cannot tell whether a pull request changes a threshold or rewrites the pipeline, because both are Python diffs.
- Onboarding is a project. A new model means a new fork, a new job definition and a week of wiring, instead of an afternoon.
The usual response is a better template repo. That helps on day one and decays by month three, because a template is copied once and then free to drift. What we want is the opposite: the pipeline is referenced, never copied.
The reframe: shared by default, override by exception
Split a model into two things:
- What the model is: identity, source table, features, algorithm, hyperparameters, gates, scoring scale, drift thresholds. This is data.
- How the model runs: the lifecycle steps. This is code, and it is identical for every model.
So the model becomes a config.yml, and the lifecycle becomes a set of model-agnostic shared notebooks that read that config at runtime. Each job passes one parameter, model_config: <model>, and the shared steps do the rest.

Tuning a model is now a config edit, never a notebook edit. If a model genuinely needs different logic for one step, it can override that single step with its own notebook. That is a deliberate, recorded decision, not the default.
What lives where
shared/notebooks/ml/ # the canonical pipeline, model-agnostic
00_setup_tables 01_data_prep 02_feature_engineering 03_model_training
04_model_evaluation 05_model_approval 06_model_deployment
07_model_scoring 08_model_monitoring 09_trigger_retraining 99_flag_for_rebuild
shared/notebooks/utils/ # config_loader, model_utils, etl_logger, repro
models/<model>/config.yml # the single source of truth for that model
models/<model>/resources/ # that model's job set (training, deployment, scoring)
models/<model>/notebooks/ # ONLY the steps this model overrides
control_plane/registry.yml # inventory: every model, owner, status, champion version
schema/model_config.schema.json
A slice of a real config shows how much a model is expressed as data:
model:
name: application_scorecard
use_case: pl_scorecard # Unity Catalog schema for this model family
use_case_prefix: pl # unique prefix for job names and resource keys
source_table: pl_application_scorecard_data
target: target_flag
primary_key: application_id
feature_eng:
encoding: dummies # banded one-hot inputs, classic scorecard
iv_selected: 0.02
banding:
age: { cutoffs: [25, 30, 36, 45] }
income: { cutoffs: [4000] }
training:
engine: statsmodels_scorecard # or sklearn: lr, xgboost, random_forest...
explainability: coefficients
reject_inference: true
metric_gate: { name: auc, min: 0.55 }
scoring:
target_score: 600
target_odds: 50
pdo: 20
monitoring:
psi_warning: 0.10
psi_critical: 0.25Swapping logistic regression for XGBoost with an Optuna search is a config change (algorithm: xgboost, plus a tuning block), not a fork. Only genuinely different logic justifies an override:
pipeline:
model_evaluation: custom # use models/<m>/notebooks/ml/04_model_evaluation.py
The lifecycle: three jobs, two gates
Every model gets the same three Databricks jobs, deployed as code.
Job Steps What it produces
| Training | data prep → feature engineering → training | A new model version registered in Unity Catalog with the @Challenger alias, only if it clears the metric gate |
| Deployment | evaluation → approval → deployment | @Challenger promoted to @Champion if it beats the live one. Otherwise a rebuild request is logged |
| Scoring | scoring → monitoring → retraining trigger | Scores, scaled points and risk grades, plus PSI/VSI drift. Critical drift starts a new training run |
The two gates are what make this safe to automate:
- Gate 1, the metric gate. Training will not register a Challenger below the configured minimum (AUC, Gini or KS on the holdout sample). Below it, the step fails and nothing registers.
- Gate 2, the champion gate. The evaluation step compares the Challenger to the current Champion on holdout and out-of-time samples. If it does not win, the flow takes the flag_for_rebuild branch. A full rebuild is a human decision, not an automatic one.
Closing the loop, the monitoring step computes PSI (score drift) and VSI (per-variable drift) against the training baseline. If either crosses its critical threshold, the next step triggers retraining. The model watches itself.
Onboarding a model is four actions
- Copy the baseline template for the new model (make onboard MODEL=... USE_CASE=... PREFIX=... SOURCE=...).
- Edit config.yml.
- Push to GitHub and open a pull request.
- Let the pipeline validate and deploy.
The Asset Bundle picks the new model up automatically through a glob include. Nothing else in the repository changes:
include:
- models/*/resources/*.yml # every model owns its job set, auto-discovered
A pre-init script renders each model's job names, keys, tags and owner from its config.yml, so the job files stay tokenised and the config stays the only thing you edit.
Governance that runs before anything deploys
A config-driven design only works if a bad config cannot reach production. So the repository validates itself on every pull request:
- Schema validation. Every config.yml is checked against a JSON Schema: types, patterns, allowed values.
- Cross-field rules. Things a schema alone cannot say. dev_ratio + hol_ratio must be below 1. psi_critical must exceed psi_warning. use_case_prefix must be unique across models, or job keys collide. A custom override must have its notebook on disk, and an orphaned override notebook is an error.
- Registry sync. control_plane/registry.yml lists every model with its owner, use case and status. A check keeps it in sync with the models/ folders, so "what models exist and who owns them?" always has one answer.
- Catalog guard. A deployment fails if the target environment has no Unity Catalog configured.
- Structural tests. An automated test suite checks bundle integrity, for example that every job's notebook path resolves.
If any check fails, the deploy step never runs.
From pull request to production
Nothing is deployed by hand.
Branch Purpose On merge
| feature/* | One change: a new model, a config edit, a fix | Nothing deploys |
| stg | Staging integration | CI validates, then deploys the bundle to the staging workspace |
| main | Production | CI validates, pauses for a required reviewer, deploys, then verifies the jobs exist |
A few details that earn their keep:
- No long-lived secrets. CI authenticates to Databricks with federated identity (Azure OIDC), so there is no stored token to leak or rotate.
- Deploy definitions, never run them. The release pipeline deploys job definitions only. Training and scoring run on their own schedules, so a release can never accidentally retrain a production model.
- Every deployment is traceable. The commit SHA is injected as a bundle variable (--var=git_sha=...), so any running job maps to an exact version of the code.
- Production approval is enforced twice. A human sets an approval tag on the model version in Unity Catalog, and the GitHub production environment requires a reviewer. Both must be satisfied. Staging approves automatically.
What we got, and what we would warn you about
Once the shared pipeline is proven on the first model, the rest is repeatable work that the customer's own team can run. After handover, their team onboarded around 10 further models on their own.
I would rather you hear the limits from me than find them later:
- Rollback is a manual step today. You re-point the @Champion alias to the previous version, which the deployment step records for audit. One-command rollback is on the roadmap.
- Overriding a step is a manual copy plus a config edit. The validator catches mistakes, but a helper to automate it is still planned.
- The platform's native monitoring dashboard depends on network connectivity that varies by environment. The drift metrics computed by the monitoring step stay available by direct query either way.
- This pipeline is deliberately scoped to binary-classification PD scorecards. Regression is out of scope by design. The pattern generalises. This implementation does not claim to.
The takeaway
For years the default was to treat each model as its own project and copy the last one. That quietly became the slowest, riskiest and most expensive way to run ML in a regulated business.
A config-driven blueprint changes the deal:
- A model is a config. Identity, features, algorithm, gates, scaling and thresholds live in one validated YAML file.
- The pipeline is shared. One tested lifecycle serves every model. Fixes propagate to all of them.
- Override by exception. Different logic is a deliberate, recorded fork of one step, never the default.
- Gate everything that can be gated. Schema, registry and catalog checks before deploy. Metric and champion gates before promotion. A human approval before production.
- Close the loop. Drift monitoring that triggers retraining turns a deployed model into a maintained one.
The fastest model to onboard is the one that needs no new code.
Next up: how we used this same framework to move a large SAS risk-model estate onto Databricks, and how we proved the results matched.
References
- Databricks Asset Bundles, Databricks documentation
- MLflow Model Registry on Unity Catalog (model aliases), Databricks documentation
- Databricks Workflows / Lakeflow Jobs, Databricks documentation