topic Re: DLT Dedupping Best Practice in Medallion in Data Engineering

DLT Dedupping Best Practice in Medallion

ChristianRRL — Wed, 10 Jan 2024 20:27:45 GMT

Hi there, I have what may be a deceptively simple question but I suspect may have a variety of answers:

What is the "right" place to handle dedupping using the medallion architecture?

In my example, I already have everything properly laid out with data arriving in a `landing` location, and I even have a DLT job that can loop through all respective source CSV > target DELTA tables. At the moment, I have the data come in entirely as the raw CSVs into a bronze delta table (DLT Streaming) and there is no dedupping done whatsoever here. If the same data is sent via two differently timestamped CSV's, *all* of the data will show in bronze.

My current intent is to have all the raw data arrive in bronze, and then I'll dedup it in a second silver delta table (DLT Streaming).

Does this make sense? I'm curious if others handle this the same way, or if it is more common practice to handle dedupping in the bronze table instead?

Re: DLT Dedupping Best Practice in Medallion

cgrant — Wed, 04 Dec 2024 08:03:28 GMT

A typical recommendation is to not do any transformations as the data lands into the bronze layer (ELT). The idea is that you want your bronze layer to be as close of a representation of your source data as possible so if there are any mistakes later, or taking your example, the existence of duplicates is a helpful indicator of some kind, it's nice to have an accurate system of record.

So in your example, raw data lands in bronze as it is and is deduplicated in the silver layer. These are not hard and fast rules - they are up to your practice.

Re: DLT Dedupping Best Practice in Medallion

Sidhant07 — Mon, 09 Dec 2024 07:46:12 GMT

1. Deduplication in medallion architecture can be handled in bronze or silver layer.
2. If keeping a complete history of all raw data, including duplicates, in the bronze layer, handle deduplication in the silver layer.
3. If not keeping a complete history of all raw data, including duplicates, in the bronze layer, handle deduplication in the bronze layer to reduce data processed in the silver layer.
4. Consider data volume, performance, cost, and data quality when deciding where to handle deduplication.
5. In your use case, handling deduplication in the silver layer is valid, but consider moving it to the bronze layer if processing a large amount of duplicate data in the silver layer.