DLT File Level Deduplication

dgahram — Mon, 15 Dec 2025 19:01:20 GMT

I want to create a DLT pipeline that incrementally processes csv files arriving daily. However, some of those files are duplicate - they have the same names and data but are in different directories. What is the best way to handle this? I'm assuming that row-level deduplication would be inefficient, but not sure if file-level deduplication is possible with DLT streaming.

Re: DLT File Level Deduplication

K_Anudeep — Tue, 16 Dec 2025 02:20:20 GMT

Hello @dgahram ,

Auto Loader tracks ingestion progress by persisting discovered file metadata in a RocksDB store within the checkpoint, which provides “exactly-once” processing for discovered files.Doc: https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/
However, if the same content appears under different file paths (for example, duplicated into another directory), Auto Loader will still “discover” it as a new file and ingest it (because it is, from a file-discovery standpoint).

In DLT, there is also no option for file deduplication. The best practice is to file de-dupe before DLT ingestion. Inside streaming, “file-level dedup” is awkward because the stream is already row-oriented after CSV parsing.

topic DLT File Level Deduplication in Data Engineering

DLT File Level Deduplication

Re: DLT File Level Deduplication