cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

how to implement delta load when table only has primary columns

Puneet096
New Contributor II

I have a table where there are two columns and both are primary key, I want to do delta load when taking data from source to target. Any idea how to implement this? 

4 REPLIES 4

szymon_dybczak
Contributor

Hi @Puneet096 ,

One approach you can use is merge into statement. Probably you can delete update part from below merge template, because it's unlikely that there will be update of values in composite primary key

 

 

MERGE INTO target_table AS target
USING source_table AS source
ON target.pk1 = source.pk1 AND target.pk2 = source.pk2
WHEN MATCHED AND source.last_updated > target.last_updated THEN
  UPDATE SET
    target.data = source.data,
    target.last_updated = source.last_updated
WHEN NOT MATCHED THEN
  INSERT (pk1, pk2, data, last_updated)
  VALUES (source.pk1, source.pk2, source.data, source.last_updated);

 

 

 

Puneet096
New Contributor II

Thanks for the response @szymon_dybczak, but the table has only two columns and both are primary key

szymon_dybczak
Contributor

But that shouldn't be a problem. In merge condition you check both keys as in example above. If combination of two keysb already exists in the table then do nothing. If there is new combination of key1 and key2 just insert it into target table.

It's that simple, or maybe I don't fully understand your problem.

 

Kaniz_Fatma
Community Manager
Community Manager

Hi @Puneet096Implementing a delta load when your table has only primary key columns can be achieved using the MERGE statement in SQL.

  1. Use the MERGE statement to compare the source and target tables based on the primary key columns.
  2. Insert records that are present in the source but not in the target.
  3. Update records in the target that have changed in the source.

If you have any specific constraints or additional requirements, feel free to share them!

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group