How to merge delta data..

[Krishscientist]

Data from Parquet to delta converted and delta files written into diff folders based on SRC_SYS_ID....

Any one help me how to merge delta data from multiple folders.

Regards.

[Aashita]

@Krishna Kommineni​ ,

MERGE command allows you to perform “upserts”, which are a mix of an UPDATE and an INSERT

To understand upserts, imagine that you have an existing table (a.k.a. a target table), and a source table that contains a mix of new records and updates to existing records. Here’s how an upsert works:

• When a record from the source table matches a preexisting record in the target table, Delta Lake updates the record.
• When there is no such match, Delta Lake inserts the new record.

Example code-

MERGE INTO events
USING updates
    ON events.eventId = updates.eventId
    WHEN MATCHED THEN UPDATE
        SET events.data = updates.data
    WHEN NOT MATCHED THEN 
        INSERT (date, eventId, data) VALUES (date, eventId, data)

[Kaniz]

Hi @Krishna Kommineni​  , Just a friendly follow-up. Do you still need help, or @Aashita Ramteke​'s response help you to find the solution? Please let us know.

[Noopur_Nigam]

Hi @Krishna Kommineni​ Is the table partitioned on SRC_SYS_ID col?

[Kaniz]

Hi @Krishna Kommineni​ ​, We haven’t heard from you on the last response from @Noopur Nigam​ , and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community as it can be helpful to others. Otherwise, we will respond with more details and try to help.