cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to facilitate incremental updates to an SCD Type 1 table that uses SCD Type 2 source tables

mvmiller
New Contributor III

I have an SCD Type 1 delta table (target) for which I am trying to figure out how to facilitate insert, updates, and deletes.  This table is sourced by multiple delta tables, with an SCD Type 2 structure, which are joined together to create the target table.  The goal is to implement an incremental load, such that 1) only records in the target table that are impacted by changes to the source tables are updated or deleted, and 2) only new records from the source that are not currently in the target table are inserted.

Each of the SCD Type 2 source tables are updated independently of one another. So it's possible that no tables, one table, or several source tables may have incremented versions since the prior time a load to the target table occurred.

I am trying to think through an process for how to facilitate incremental updates to the target table that does not require a full reload of the table.  Thoughts and advice or much appreciated.

1 REPLY 1

mvmiller
New Contributor III

Correction (I can't seem to edit or remove original post):

- "... trying to think through an process" --> *a* process

- "Thoughts and advice or much appreciated" --> Thoughts and/or advice are much appreciated.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group