Problem

Duplicate or slightly varied entity records (names, emails, phones, addresses) across systems lead to:

• Poor analytics and reporting
• Compliance and audit risks
• Heavy manual review effort that doesn’t scale

High‑level approach on Databricks

Here’s the pattern I’ve been following — intentionally kept modular and configurable so it can be reused across domains:

1. Normalization & validation

   • Standardize names, emails, phones, and addresses
   • Persist curated Delta tables with lineage/audit metadata

2. Blocking strategy

   • Reduce candidate pairs using lightweight keys (e.g., phonetic variants, email domain–style grouping)
   • Focus on scalability before expensive comparisons

3. Fuzzy matching & scoring

   • Weighted similarity scores across name, email, address
   • Generate “potential match” pairs instead of binary decisions

4. Clustering

   • Group related records using a connected‑components style approach to form entity clusters

5. Golden record selection

   • Rule‑based selection driven by completeness, recency, and data quality signals

6. ML‑assisted matching

   • Train a simple classifier on engineered similarity features
   • Register and manage the model with MLflow and use it to auto‑classify high‑confidence matches

7. Human‑in‑the‑loop review

   • Low‑confidence matches routed for manual review
   • Reviewer decisions written back to Delta tables to support auditability and future retraining

The intent is not full automation, but a balanced design that combines ML, rules, and supervised review — especially important in regulated environments.

What I’d love input on from the community

• What blocking strategies have worked well for you at scale, especially for name + address heavy datasets?
• How do you typically decide thresholds between:
  • auto‑merge
  • manual review
  • no‑match
• Any best practices for storing reviewer decisions so they’re useful for both audit and model improvement?

If there’s interest, I’m happy to follow up with a deeper breakdown (pseudo‑code, table design, orchestration patterns, etc.).

Looking forward to learning from your experiences — thanks in advance! 🙌