How to merge delta data..

Krishscientist — Fri, 13 May 2022 22:00:52 GMT

Data from Parquet to delta converted and delta files written into diff folders based on SRC_SYS_ID....

Any one help me how to merge delta data from multiple folders.

Regards.

Re: How to merge delta data..

Aashita — Sat, 14 May 2022 06:40:10 GMT

@Krishna Kommineni ,

MERGE command allows you to perform “upserts”, which are a mix of an UPDATE and an INSERT

To understand upserts, imagine that you have an existing table (a.k.a. a target table), and a source table that contains a mix of new records and updates to existing records. Here’s how an upsert works:

When a record from the source table matches a preexisting record in the target table, Delta Lake updates the record.
When there is no such match, Delta Lake inserts the new record.

Example code-

MERGE INTO events
USING updates
    ON events.eventId = updates.eventId
    WHEN MATCHED THEN UPDATE
        SET events.data = updates.data
    WHEN NOT MATCHED THEN 
        INSERT (date, eventId, data) VALUES (date, eventId, data)

Re: How to merge delta data..

Noopur_Nigam — Thu, 02 Jun 2022 02:10:02 GMT

Hi @Krishna Kommineni Is the table partitioned on SRC_SYS_ID col?

topic Re: How to merge delta data.. in Data Engineering

How to merge delta data..

Re: How to merge delta data..

Re: How to merge delta data..