Sidhant07
Databricks Employee
Databricks Employee

1. Deduplication in medallion architecture can be handled in bronze or silver layer.
2. If keeping a complete history of all raw data, including duplicates, in the bronze layer, handle deduplication in the silver layer.
3. If not keeping a complete history of all raw data, including duplicates, in the bronze layer, handle deduplication in the bronze layer to reduce data processed in the silver layer.
4. Consider data volume, performance, cost, and data quality when deciding where to handle deduplication.
5. In your use case, handling deduplication in the silver layer is valid, but consider moving it to the bronze layer if processing a large amount of duplicate data in the silver layer.