topic Near Real time Solutioning on data from Core System which gets updated in Data Engineering

Near Real time Solutioning on data from Core System which gets updated

Kroy — Fri, 15 Dec 2023 04:56:19 GMT

We are trying to build solution , where customer data stored in one of RDBM database SQL server and we are moving this data to delta lake in medallion architecture and want to this to be near real time by using DLT pipeline.

Problem is that source tables rows gets updated , so our bronze delta table will also behave same ,so we can 't create Live table on top of this due to bronze table being updated.

What could be the solution ?

Re: Near Real time Solutioning on data from Core System which gets updated

Kroy — Fri, 15 Dec 2023 05:20:47 GMT

Hi @Retired_mod ,

Your Solution are to update the Live table, but my problem is how can I create the Live table from the delta table which always gets updated.

I believe we can only create live table from stream table which only appends ?

Re: Near Real time Solutioning on data from Core System which gets updated

Kroy — Thu, 08 Feb 2024 13:59:27 GMT

came across this matrix while reading about DLT

what is read from complete and write to Incremental means ?