Data Drift Metrics

[suryaprayaga]

As part of flawless data governance and data quality implementation, Data Drift metrics play a very important role. I have compiled a few Data Drift metrics like Jensen-Shannon Divergence Score, Correlation Breakdown Detection, Data Lineage Health and Blast Radius Score etc., 

If there are multiple features available in your dataset, it's important to measure the Jensen-Shannon Divergence score feature by feature and combine the individual JSD scores into an overall Population Drift Score. 

What other innovative & novel KPIs did you use in your projects that measure the quality of the data before being consumed in to your workloads?

These are natural discussions that go hand-in-hand if you are using DQX framework in Databricks.

[balajij8 · Accepted solution]

Hi Surya,

Statistical Drift Metrics to handle distribution shifts across numeric and categorical features

• Population Stability Index - Excellent for tracking numeric feature distribution shifts and detecting gradual concept drift over time.

• Kolmogorov-Smirnov (KS) Test - Evaluates distribution drift for numeric columns while providing statistical significance.

• Wasserstein Distance - Measures Earth Mover's Distance for numeric distributions, making it far more sensitive to tail changes than JS Divergence.

• Chi-Squared Test - Tracks categorical distribution shifts along with $p$-value significance.

• Total Variation Distance - Computes the $L_1$ distance across categorical distributions.

• L-Infinity Distance - Pinpoints the single maximum difference across categorical bins.

Operational Metrics to tracking before data hits downstream workloads to prevent failures

• Freshness Score - Measures the time elapsed since the last update, weighted against expected update frequency and downstream dependencies.

• Completeness Rate - Calculates non-null percentages across critical columns, aggregated with business priority weights rather than treating all columns equally.

• Schema Evolution - Tracks the rate and frequency of schema alterations to assess pipeline and source stability.

• Downstream Impact - Multiplies the number of affected downstream assets by their execution and query frequency (dashboards, jobs, ad-hoc queries).

Advanced Drift to catch subtle issues across complex multi-feature datasets and ML pipelines

• Multi-variate Drift - Combines individual feature drift scores into a single composite metric using weighted aggregation or PCA-based distance.

• Correlation Stability Matrix - Tracks changes in feature-to-feature correlations over rolling windows to spot structural breaks.

• Prediction Drift vs Label Drift - For ML inference datasets, explicitly separates model output drift from true ground-truth label drift.

• Segment-specific Drift - Runs slice-based drift calculations on key business sub-populations (tier-1 customers, specific regions) where global metrics might mask localized drift.

Lineage-Driven Quality checks with Unity Catalog lineage

• Domain Consistency - Verifies matching entity counts across domain boundaries prior to ingestion

[suryaprayaga]

Balaji, this is an amazing list. I will use a few of them in my POC to start with. Great reply.