Hi Surya,

Statistical Drift Metrics to handle distribution shifts across numeric and categorical features

• Population Stability Index - Excellent for tracking numeric feature distribution shifts and detecting gradual concept drift over time.

• Kolmogorov-Smirnov (KS) Test - Evaluates distribution drift for numeric columns while providing statistical significance.

• Wasserstein Distance - Measures Earth Mover's Distance for numeric distributions, making it far more sensitive to tail changes than JS Divergence.

• Chi-Squared Test - Tracks categorical distribution shifts along with $p$-value significance.

• Total Variation Distance - Computes the $L_1$ distance across categorical distributions.

• L-Infinity Distance - Pinpoints the single maximum difference across categorical bins.

Operational Metrics to tracking before data hits downstream workloads to prevent failures

• Freshness Score - Measures the time elapsed since the last update, weighted against expected update frequency and downstream dependencies.

• Completeness Rate - Calculates non-null percentages across critical columns, aggregated with business priority weights rather than treating all columns equally.

• Schema Evolution - Tracks the rate and frequency of schema alterations to assess pipeline and source stability.

• Downstream Impact - Multiplies the number of affected downstream assets by their execution and query frequency (dashboards, jobs, ad-hoc queries).

Advanced Drift to catch subtle issues across complex multi-feature datasets and ML pipelines

• Multi-variate Drift - Combines individual feature drift scores into a single composite metric using weighted aggregation or PCA-based distance.

• Correlation Stability Matrix - Tracks changes in feature-to-feature correlations over rolling windows to spot structural breaks.

• Prediction Drift vs Label Drift - For ML inference datasets, explicitly separates model output drift from true ground-truth label drift.

• Segment-specific Drift - Runs slice-based drift calculations on key business sub-populations (tier-1 customers, specific regions) where global metrics might mask localized drift.

Lineage-Driven Quality checks with Unity Catalog lineage

• Domain Consistency - Verifies matching entity counts across domain boundaries prior to ingestion