Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-21-2023 02:51 PM - edited 12-21-2023 02:51 PM
I think you can explore DLT API "Apply Changes".
You can run it only in DLT pipeline but it can read from streming endpoint or streaming table.
Please check docs:
https://docs.databricks.com/en/delta-live-tables/cdc.html#language-python
You just include this line of code:
stored_as_scd_type = "2"
and you have your SCD2 logic done 🙂
But if you want to do clasic engineering you are right, this is MERGE with
whenMatchedUpdate
You can do it either in pysaprk or spraksql and you can do it also in delta table straming.
Please let me know if that helped or maybe I missunderstood your question.