Near Real time Solutioning on data from Core System which gets updated
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-14-2023 08:56 PM
We are trying to build solution , where customer data stored in one of RDBM database SQL server and we are moving this data to delta lake in medallion architecture and want to this to be near real time by using DLT pipeline.
Problem is that source tables rows gets updated , so our bronze delta table will also behave same ,so we can 't create Live table on top of this due to bronze table being updated.
What could be the solution ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-14-2023 09:20 PM
Hi @Retired_mod ,
Your Solution are to update the Live table, but my problem is how can I create the Live table from the delta table which always gets updated.
I believe we can only create live table from stream table which only appends ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-08-2024 05:59 AM
came across this matrix while reading about DLT
what is read from complete and write to Incremental means ?

