Incremental updates in Delta Live Tables

[morganmazouchi]

What happens if we change the logic for the delta live tables and we do an incremental update. Does the table get reset (refresh) automatically or would it only apply the logic to new incoming data? would we have to trigger a reset in this case?

[-werners-]

I doubt the table gets reset, as that would mean you could never change anything in a current setup.

Delta live was created to make life easier, so my guess is that it is new data only.

[Hubert-Dudek]

Delta is transactional so nothing will be reset, it can be deleted but you can always back to past using time capsule travel 🙂

One thing to which maybe you are referring is partitioning, when you overwrite data it will overwrite all partitions but you can use dynamic overwrite so you will only overwrite partitions which have new data:

spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")

[morganmazouchi · Accepted solution]

Here is my finding on when to refresh (reset) the table:

If it is a complete table all the changes would be apply automatically. 

If the table is incremental table, you need to do a manually reset (full refresh).

[jose_gonzalez]

Hi @Mojgan Mazouchi​ ,

According to the docs "Tables can be incremental or complete. Incremental tables support updates based on continually arriving data without having to recompute the entire table. A complete table is entirely recomputed with each update."

Docs here https://docs.databricks.com/data-engineering/delta-live-tables/delta-live-tables-user-guide.html#dat...

[morganmazouchi]

That implies that if there is any logic change for complete tables since it is recomputed on every update it is safe to not refresh the pipeline, where as with incremental tables if there is any change to recompute the changes we should refresh the pipeline, right?

[jose_gonzalez]

According to the docs, yes.

[morganmazouchi]

Thanks Jose!