topic loading incremental data in Data Engineering

loading incremental data

Zara — Sat, 29 Apr 2023 17:31:45 GMT

I want to load incremental data to the delta live table, I wrote function to load data for 10 tables, every time that I run the pipe line, some tables are empty and have a schema, and when I run again, the other tables are empty and the previous tables are full, it happens every time that I run the pipeline. Inside the function, I am using @dlt.table.

Re: loading incremental data

Rishabh-Pandey — Mon, 01 May 2023 10:46:47 GMT

you are loading the data in your delta live tables , there is the concept of full refresh and and refresh , if you do the full refresh it will truncate the previous data and only give you the schema , so maybe in that case you have to do the refresh only for those tables in which you dont want to do truncate. check you are doing full refresh or refresh for the tables

Re: loading incremental data

Annapurna_Hiriy — Mon, 01 May 2023 16:26:43 GMT

@zahra Jalilpour

How the DLT tables and views are updated depends on the update type:

Refresh all: All live tables are updated to reflect the current state of their input data sources. For all streaming tables, new rows are appended to the table.
Full refresh all: All live tables are updated to reflect the current state of their input data sources. For all streaming tables, Delta Live Tables attempts to clear all data from each table and then load all data from the streaming source.

For existing live tables, an update has the same behavior as a SQL REFRESH on a materialized view. For new live tables, the behavior is the same as a SQL CREATE operation.