I doubt the table gets reset, as that would mean you could never change anything in a current setup.

Delta live was created to make life easier, so my guess is that it is new data only.

Delta is transactional so nothing will be reset, it can be deleted but you can always back to past using time capsule travel 🙂

One thing to which maybe you are referring is partitioning, when you overwrite data it will overwrite all partitions but you can use dynamic overwrite so you will only overwrite partitions which have new data:

spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")

Hi @Mojgan Mazouchi​ ,

According to the docs "Tables can be incremental or complete. Incremental tables support updates based on continually arriving data without having to recompute the entire table. A complete table is entirely recomputed with each update."

Docs here https://docs.databricks.com/data-engineering/delta-live-tables/delta-live-tables-user-guide.html#dat...

That implies that if there is any logic change for complete tables since it is recomputed on every update it is safe to not refresh the pipeline, where as with incremental tables if there is any change to recompute the changes we should refresh the pipeline, right?

According to the docs, yes.

Thanks Jose!