Near Real time Solutioning on data from Core System which gets updated

[Kroy]

We are trying to build solution , where customer data stored in one of RDBM database SQL server and we are moving this data to delta lake in medallion architecture and want to this to be near real time by using DLT pipeline.

Problem is that source tables rows gets updated , so our bronze delta table  will also behave same ,so we can 't create Live table on top of this due to bronze table being updated.

What could be the solution ? 

[Kaniz_Fatma]

Hi @Kroy, Please let us know if any of these solution works in your case.

1. Delta Live Tables Pipeline Update: You can run an update on your Delta Live Tables pipeline. This update starts a cluster with the correct configuration, discovers all the tables and views defi.... There aYou can performl types of updates, such as ‘Refresh all’, ‘Full refresh all’, ‘Refresh select....
2. MERGE INTO Command: You can use the MERGE INTO command to implement your logic using Delta Live Tabl.... This command allows you to update or insert data into a Delta table conditionally and is particularl....
3. Custom Upsert Using apply_changes: You can achieve the desired behaviour using apply_changes in Data.... You can use the merge operation to merge data from your source into your target Delta table and then....
4. Updating Changed Target Rows: This approach requires upsert (merge) functionality to update the chan....

[Kroy]

Hi @Kaniz_Fatma ,

Your Solution are to update the Live table, but my problem is how can I create the Live table from the delta table which always gets updated.

I believe we can only create live table from stream table which only appends ?

[Kaniz_Fatma]

Hi @Kroy, Delta Live Tables can indeed be created from Delta tables that are updated regularly.

The Delta Live Tables runtime automatically creates tables in the Delta format and ensures those tab...

You can define datasets (tables and views) in Delta Live Tables against any query that returns a Spa.... Databricks recommends using streaming tables for most use cases for data ingestion tasks.

So, while streaming tables are commonly used with Delta Live Tables, they are not the only option. You can create a Delta Live Table from any Delta table, whether it’s being appended to, updated, or otherwise modified.

I hope this helps! Let me know if you have any other questions.

[Kroy]

came across this matrix while reading about DLT 

what is read from complete and write to Incremental means ?