cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Near Real time Solutioning on data from Core System which gets updated

Kroy
Contributor

We are trying to build solution , where customer data stored in one of RDBM database SQL server and we are moving this data to delta lake in medallion architecture and want to this to be near real time by using DLT pipeline.

Problem is that source tables rows gets updated , so our bronze delta table  will also behave same ,so we can 't create Live table on top of this due to bronze table being updated.

What could be the solution ? 

 

 

 

 

2 REPLIES 2

Kroy
Contributor

Hi @Retired_mod ,

Your Solution are to update the Live table, but my problem is how can I create the Live table from the delta table which always gets updated.

I believe we can only create live table from stream table which only appends ?

 

 

Kroy
Contributor

came across this matrix while reading about DLT 

Kroy_0-1707400734210.png

what is read from complete and write to Incremental means ?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group