Hello @dgahram ,

• Auto Loader tracks ingestion progress by persisting discovered file metadata in a RocksDB store within the checkpoint, which provides “exactly-once” processing for discovered files.Doc: https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/

• However, if the same content appears under different file paths (for example, duplicated into another directory), Auto Loader will still “discover” it as a new file and ingest it (because it is, from a file-discovery standpoint).

In DLT, there is also no option for file deduplication. The best practice is to file de-dupe before DLT ingestion. Inside streaming, “file-level dedup” is awkward because the stream is already row-oriented after CSV parsing.