cancel
Showing results for 
Search instead for 
Did you mean: 
Community Articles
Dive into a collaborative space where members like YOU can exchange knowledge, tips, and best practices. Join the conversation today and unlock a wealth of collective wisdom to enhance your experience and drive success.
cancel
Showing results for 
Search instead for 
Did you mean: 

Models Should Be Config, Not Copies: A Config-Driven MLOps Blueprint on Databricks

SrinuB
New Contributor III

Banking · Insurance · Credit Risk · Any Team Running More Than One Model — From Copy-Pasted Pipelines to One Shared Pipeline and One YAML per Model

By @SrinuB  in collaboration with @sridharplv 

Your first model went to production beautifully. Your fifth model is a copy of the first, with three notebooks edited and a job YAML nobody remembers changing.

Every ML platform I have worked on follows the same arc. The first model gets a careful pipeline. The second gets a copy of it. By the fifth, there are five slightly different versions of "the" training notebook, a bug fix that landed in two of them, and a promotion step that works differently depending on who wrote it. That drift is where the 2 AM incidents and the audit findings come from. It is rarely the model itself.

This article is about removing that drift. I'll walk through the blueprint we built for a large financial institution's credit-risk modelling team, where a model is a single config.yml file and everything else is a shared, tested pipeline. The worked example is a PD (probability of default) scorecard, but the pattern applies to any team that runs more than a handful of models.

The problem: every model becomes its own snowflake

Strip away the industry and the pattern looks like this. A model needs the same lifecycle every time: prepare data, engineer features, train, evaluate against what is live, get approved, deploy, score, monitor, retrain. What differs between models is a small set of decisions: which table, which target, which algorithm, which thresholds.

When those decisions are expressed as code in a copied notebook, four things go wrong at once:

  1. Fixes don't propagate. A bug fixed in model A's evaluation step stays in models B to E until someone notices.
  2. Promotion logic diverges. One model auto-promotes, another needs a manual alias flip, a third has a check nobody can find.
  3. Reviews get expensive. A reviewer cannot tell whether a pull request changes a threshold or rewrites the pipeline, because both are Python diffs.
  4. Onboarding is a project. A new model means a new fork, a new job definition and a week of wiring, instead of an afternoon.

The usual response is a better template repo. That helps on day one and decays by month three, because a template is copied once and then free to drift. What we want is the opposite: the pipeline is referenced, never copied.

The reframe: shared by default, override by exception

Split a model into two things:

  • What the model is: identity, source table, features, algorithm, hyperparameters, gates, scoring scale, drift thresholds. This is data.
  • How the model runs: the lifecycle steps. This is code, and it is identical for every model.

So the model becomes a config.yml, and the lifecycle becomes a set of model-agnostic shared notebooks that read that config at runtime. Each job passes one parameter, model_config: <model>, and the shared steps do the rest.

blog1_mlops_blueprint_diagram.png

Tuning a model is now a config edit, never a notebook edit. If a model genuinely needs different logic for one step, it can override that single step with its own notebook. That is a deliberate, recorded decision, not the default.

What lives where

shared/notebooks/ml/        # the canonical pipeline, model-agnostic
  00_setup_tables  01_data_prep  02_feature_engineering  03_model_training
  04_model_evaluation  05_model_approval  06_model_deployment
  07_model_scoring  08_model_monitoring  09_trigger_retraining  99_flag_for_rebuild
shared/notebooks/utils/     # config_loader, model_utils, etl_logger, repro
models/<model>/config.yml   # the single source of truth for that model
models/<model>/resources/   # that model's job set (training, deployment, scoring)
models/<model>/notebooks/   # ONLY the steps this model overrides
control_plane/registry.yml  # inventory: every model, owner, status, champion version
schema/model_config.schema.json

A slice of a real config shows how much a model is expressed as data:

model:
  name: application_scorecard
  use_case: pl_scorecard            # Unity Catalog schema for this model family
  use_case_prefix: pl               # unique prefix for job names and resource keys
  source_table: pl_application_scorecard_data
  target: target_flag
  primary_key: application_id

feature_eng:
  encoding: dummies                 # banded one-hot inputs, classic scorecard
  iv_selected: 0.02
  banding:
    age:    { cutoffs: [25, 30, 36, 45] }
    income: { cutoffs: [4000] }

training:
  engine: statsmodels_scorecard     # or sklearn: lr, xgboost, random_forest...
  explainability: coefficients
  reject_inference: true
  metric_gate: { name: auc, min: 0.55 }

scoring:
  target_score: 600
  target_odds: 50
  pdo: 20

monitoring:
  psi_warning: 0.10
  psi_critical: 0.25

Swapping logistic regression for XGBoost with an Optuna search is a config change (algorithm: xgboost, plus a tuning block), not a fork. Only genuinely different logic justifies an override:

pipeline:
  model_evaluation: custom   # use models/<m>/notebooks/ml/04_model_evaluation.py

The lifecycle: three jobs, two gates

Every model gets the same three Databricks jobs, deployed as code.

Job Steps What it produces

Trainingdata prep → feature engineering → trainingA new model version registered in Unity Catalog with the @Challenger alias, only if it clears the metric gate
Deploymentevaluation → approval → deployment@Challenger promoted to @Champion if it beats the live one. Otherwise a rebuild request is logged
Scoringscoring → monitoring → retraining triggerScores, scaled points and risk grades, plus PSI/VSI drift. Critical drift starts a new training run

The two gates are what make this safe to automate:

  • Gate 1, the metric gate. Training will not register a Challenger below the configured minimum (AUC, Gini or KS on the holdout sample). Below it, the step fails and nothing registers.
  • Gate 2, the champion gate. The evaluation step compares the Challenger to the current Champion on holdout and out-of-time samples. If it does not win, the flow takes the flag_for_rebuild branch. A full rebuild is a human decision, not an automatic one.

Closing the loop, the monitoring step computes PSI (score drift) and VSI (per-variable drift) against the training baseline. If either crosses its critical threshold, the next step triggers retraining. The model watches itself.

Onboarding a model is four actions

  1. Copy the baseline template for the new model (make onboard MODEL=... USE_CASE=... PREFIX=... SOURCE=...).
  2. Edit config.yml.
  3. Push to GitHub and open a pull request.
  4. Let the pipeline validate and deploy.

The Asset Bundle picks the new model up automatically through a glob include. Nothing else in the repository changes:

include:
  - models/*/resources/*.yml   # every model owns its job set, auto-discovered

A pre-init script renders each model's job names, keys, tags and owner from its config.yml, so the job files stay tokenised and the config stays the only thing you edit.

Governance that runs before anything deploys

A config-driven design only works if a bad config cannot reach production. So the repository validates itself on every pull request:

  • Schema validation. Every config.yml is checked against a JSON Schema: types, patterns, allowed values.
  • Cross-field rules. Things a schema alone cannot say. dev_ratio + hol_ratio must be below 1. psi_critical must exceed psi_warning. use_case_prefix must be unique across models, or job keys collide. A custom override must have its notebook on disk, and an orphaned override notebook is an error.
  • Registry sync. control_plane/registry.yml lists every model with its owner, use case and status. A check keeps it in sync with the models/ folders, so "what models exist and who owns them?" always has one answer.
  • Catalog guard. A deployment fails if the target environment has no Unity Catalog configured.
  • Structural tests. An automated test suite checks bundle integrity, for example that every job's notebook path resolves.

If any check fails, the deploy step never runs.

From pull request to production

Nothing is deployed by hand.

Branch Purpose On merge

feature/*One change: a new model, a config edit, a fixNothing deploys
stgStaging integrationCI validates, then deploys the bundle to the staging workspace
mainProductionCI validates, pauses for a required reviewer, deploys, then verifies the jobs exist

A few details that earn their keep:

  • No long-lived secrets. CI authenticates to Databricks with federated identity (Azure OIDC), so there is no stored token to leak or rotate.
  • Deploy definitions, never run them. The release pipeline deploys job definitions only. Training and scoring run on their own schedules, so a release can never accidentally retrain a production model.
  • Every deployment is traceable. The commit SHA is injected as a bundle variable (--var=git_sha=...), so any running job maps to an exact version of the code.
  • Production approval is enforced twice. A human sets an approval tag on the model version in Unity Catalog, and the GitHub production environment requires a reviewer. Both must be satisfied. Staging approves automatically.

What we got, and what we would warn you about

Once the shared pipeline is proven on the first model, the rest is repeatable work that the customer's own team can run. After handover, their team onboarded around 10 further models on their own.

I would rather you hear the limits from me than find them later:

  • Rollback is a manual step today. You re-point the @Champion alias to the previous version, which the deployment step records for audit. One-command rollback is on the roadmap.
  • Overriding a step is a manual copy plus a config edit. The validator catches mistakes, but a helper to automate it is still planned.
  • The platform's native monitoring dashboard depends on network connectivity that varies by environment. The drift metrics computed by the monitoring step stay available by direct query either way.
  • This pipeline is deliberately scoped to binary-classification PD scorecards. Regression is out of scope by design. The pattern generalises. This implementation does not claim to.

The takeaway

For years the default was to treat each model as its own project and copy the last one. That quietly became the slowest, riskiest and most expensive way to run ML in a regulated business.

A config-driven blueprint changes the deal:

  1. A model is a config. Identity, features, algorithm, gates, scaling and thresholds live in one validated YAML file.
  2. The pipeline is shared. One tested lifecycle serves every model. Fixes propagate to all of them.
  3. Override by exception. Different logic is a deliberate, recorded fork of one step, never the default.
  4. Gate everything that can be gated. Schema, registry and catalog checks before deploy. Metric and champion gates before promotion. A human approval before production.
  5. Close the loop. Drift monitoring that triggers retraining turns a deployed model into a maintained one.

The fastest model to onboard is the one that needs no new code.

Next up: how we used this same framework to move a large SAS risk-model estate onto Databricks, and how we proved the results matched.

References

  • Databricks Asset Bundles, Databricks documentation
  • MLflow Model Registry on Unity Catalog (model aliases), Databricks documentation
  • Databricks Workflows / Lakeflow Jobs, Databricks documentation
0 REPLIES 0