Hi Community,

We are currently designing a Data Harmonization framework in Databricks and would appreciate insights from anyone who has implemented something similar at scale.

Context:
We are ingesting data from multiple source systems where:
- Different sources provide similar business objects
- Each source has different schemas and naming conventions
- Data types and formats vary
- There may be cross-source conflicts in attribute values

Harmonization will be implemented in the Silver Layer, creating a dedicated:
Silver → Harmonized_Silver Layer

Key Design Considerations:
We want the solution to be configuration-driven and reusable, not hardcoded per object.

The framework should support:
- Data Harmonization from Different Source Systems
- Handle different objects across multiple sources
- Support schema variability
- Harmonization in Silver Layer
- Transform curated Silver data into a standardized Harmonized_Silver model

Standard Harmonization Rules (Driven by Configuration)
- Similar object merging
- Column mapping via metadata/config tables
- Schema standardization across source systems
- Data type and format normalization
- Enforce data quality rules
- Resolve cross-source conflicts (priority rules, survivorship logic, etc.)
- Maintain full auditability and lineage

Create a Generic Data Harmonization Model
We are aiming to design a reusable harmonization model that:
- Works across domains (Customer, Product, Order, etc.)
- Supports schema evolution
- Supports incremental loads
- Is scalable for large datasets (100M+ records)
- Maintains traceability to source systems

Questions for the Community:
- Has anyone implemented a similar config-driven harmonization model in Databricks?
- What architecture worked best (Delta Live Tables vs structured jobs/notebooks)?
- How did you handle cross-source conflict resolution logic at scale?
- What is the best approach for maintaining lineage and auditability (Unity Catalog, custom audit tables, etc.)?
- Any performance challenges or anti-patterns to avoid?

We are targeting an enterprise-grade design and would greatly appreciate any best practices, patterns, or lessons learned.

Thank you.