cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Near Real time Solutioning on data from Core System which gets updated

Kroy
Contributor

We are trying to build solution , where customer data stored in one of RDBM database SQL server and we are moving this data to delta lake in medallion architecture and want to this to be near real time by using DLT pipeline.

Problem is that source tables rows gets updated , so our bronze delta table  will also behave same ,so we can 't create Live table on top of this due to bronze table being updated.

What could be the solution ? 

 

 

 

 

2 REPLIES 2

Kroy
Contributor

Hi @Retired_mod ,

Your Solution are to update the Live table, but my problem is how can I create the Live table from the delta table which always gets updated.

I believe we can only create live table from stream table which only appends ?

 

 

Kroy
Contributor

came across this matrix while reading about DLT 

Kroy_0-1707400734210.png

what is read from complete and write to Incremental means ?

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now