topic Re: How to Prevent Duplicate Entries to enter to delta lake of Azure Storage in Data Engineering

How to Prevent Duplicate Entries to enter to delta lake of Azure Storage

User16826994223 — Fri, 11 Jun 2021 15:27:59 GMT

I Have a Dataframe stored in the format of delta into Adls, now when im trying to append new updated rows to that delta lake it should, Is there any way where i can delete the old existing record in delta and add the new updated Record.

There is a unique Column for the schema of DataFrame stored in Delta. by which we can check whether the record is updated or new.

Re: How to Prevent Duplicate Entries to enter to delta lake of Azure Storage

Ryan_Chynoweth — Fri, 18 Jun 2021 21:44:32 GMT

You should use a MERGE command on this table to match records on the unique column. Delta Lake does not enforce primary keys so if you append only the duplicate ids will appear.

Merge will provide you the functionality you desire.

https://docs.databricks.com/spark/latest/spark-sql/language-manual/delta-merge-into.html

Re: How to Prevent Duplicate Entries to enter to delta lake of Azure Storage

652852 — Tue, 04 Oct 2022 06:10:26 GMT

According to the documentation, COPY INTO is supposed to be idempotent, and on successive runs, it shouldn't be reloading already loaded files. In my case, I created a table from existing data in S3 (many files). Then, hoping to load only newly arrived files (batch ingestion), I tried COPY INTO, but it went ahead and naively reloaded everything from S3.

I also tried with MERGE, but it looks like the source can't be parquet files in S3, it can only be similarly a Delta Table?