topic Incremental updates in Delta Live Tables in Data Engineering

Incremental updates in Delta Live Tables

morganmazouchi — Sun, 28 Nov 2021 19:31:48 GMT

What happens if we change the logic for the delta live tables and we do an incremental update. Does the table get reset (refresh) automatically or would it only apply the logic to new incoming data? would we have to trigger a reset in this case?

Re: Incremental updates in Delta Live Tables

-werners- — Mon, 29 Nov 2021 07:34:25 GMT

I doubt the table gets reset, as that would mean you could never change anything in a current setup.

Delta live was created to make life easier, so my guess is that it is new data only.

Re: Incremental updates in Delta Live Tables

Hubert-Dudek — Mon, 29 Nov 2021 10:37:17 GMT

Delta is transactional so nothing will be reset, it can be deleted but you can always back to past using time capsule travel 🙂

One thing to which maybe you are referring is partitioning, when you overwrite data it will overwrite all partitions but you can use dynamic overwrite so you will only overwrite partitions which have new data:

spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")

Re: Incremental updates in Delta Live Tables

morganmazouchi — Mon, 29 Nov 2021 15:47:26 GMT

Here is my finding on when to refresh (reset) the table:

If it is a complete table all the changes would be apply automatically.

If the table is incremental table, you need to do a manually reset (full refresh).

Re: Incremental updates in Delta Live Tables

jose_gonzalez — Mon, 29 Nov 2021 18:47:37 GMT

Hi @Mojgan Mazouchi ,

According to the docs "Tables can be incremental or complete. Incremental tables support updates based on continually arriving data without having to recompute the entire table. A complete table is entirely recomputed with each update."

Docs here https://docs.databricks.com/data-engineering/delta-live-tables/delta-live-tables-user-guide.html#datasets

Re: Incremental updates in Delta Live Tables

morganmazouchi — Mon, 29 Nov 2021 18:58:16 GMT

That implies that if there is any logic change for complete tables since it is recomputed on every update it is safe to not refresh the pipeline, where as with incremental tables if there is any change to recompute the changes we should refresh the pipeline, right?

Re: Incremental updates in Delta Live Tables

jose_gonzalez — Fri, 10 Dec 2021 23:52:03 GMT

According to the docs, yes.

Re: Incremental updates in Delta Live Tables

morganmazouchi — Sun, 12 Dec 2021 18:30:13 GMT

Thanks Jose!