In DLT read_stream, we can't use ignoreChanges / ignoreDeletes. These are the configs helps to avoid the failures but it is actually ignoring the operations done on the upstream. So you need to manually perform the deletes or updates in the downstream. (Spark structured streaming supports ever growing / append only sources).
If you have use cases where the upstream can have updates / deletes and you want to pass these operations automatically to downstream you can follow the below suggested architectures in DLT. In both setup using live tables helps to handle updates / deletes from upstream.
Architecture 1:
You can use live tables to handle this. For use cases where you perform updates/deletes on the bronze table to reflect these deletes/updates in the silver table, you can create silver table as live table
Refer below diagram:
Architecture 2:
Other way to handle updates / deletes and pass through downstream is you can use DLT CDC. The CDC architecture looks something like below.
DLT bronze table --> DLT silver using CDC apply_changes --> DLT gold live table
Here silver table picks change data from bronze(updates or delete) and do necessary operations.
In both setups, if you delete/update any record in bronze table for use cases like GDPR, this delete/update will automatically flow to silver table(you no need to manually delete/update from silver and then gold). Now gold will pick this silver table and perform full refresh. (live table).
Also DLT has a special feature called enลพyme. Enลพyme helps to avoid full re-computation for the LIVE table and improve the performance.
What is enลพyme?
Compared to the existing method of fully recomputing all rows in the live table โ even rows which do not need to be changed โ enลพyme may significantly reduce resource utilization and improve overall pipeline latency by only updating the rows in the live table which are necessary to materialize the result.
For more details on enลพyme you can refer this blog: https://www.databricks.com/blog/2022/06/29/delta-live-tables-announces-new-capabilities-and-performa...