cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Near Real time Solutioning on data from Core System which gets updated

Kroy
Contributor

We are trying to build solution , where customer data stored in one of RDBM database SQL server and we are moving this data to delta lake in medallion architecture and want to this to be near real time by using DLT pipeline.

Problem is that source tables rows gets updated , so our bronze delta table  will also behave same ,so we can 't create Live table on top of this due to bronze table being updated.

What could be the solution ? 

 

 

 

 

4 REPLIES 4

Kaniz_Fatma
Community Manager
Community Manager

Kroy
Contributor

Hi @Kaniz_Fatma ,

Your Solution are to update the Live table, but my problem is how can I create the Live table from the delta table which always gets updated.

I believe we can only create live table from stream table which only appends ?

 

 

Kaniz_Fatma
Community Manager
Community Manager

Hi @KroyDelta Live Tables can indeed be created from Delta tables that are updated regularly.

The Delta Live Tables runtime automatically creates tables in the Delta format and ensures those tab...

You can define datasets (tables and views) in Delta Live Tables against any query that returns a Spa...Databricks recommends using streaming tables for most use cases for data ingestion tasks.

So, while streaming tables are commonly used with Delta Live Tables, they are not the only option. You can create a Delta Live Table from any Delta table, whether it’s being appended to, updated, or otherwise modified.

I hope this helps! Let me know if you have any other questions.

Kroy
Contributor

came across this matrix while reading about DLT 

Kroy_0-1707400734210.png

what is read from complete and write to Incremental means ?

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!