cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

loading incremental data

Zara
New Contributor II

I want to load incremental data to the delta live table, I wrote function to load data for 10 tables, every time that I run the pipe line, some tables are empty and have a schema, and when I run again, the other tables are empty and the previous tables are full, it happens every time that I run the pipeline. Inside the function, I am using @dlt.table.

2 REPLIES 2

Rishabh264
Honored Contributor II

you are loading the data in your delta live tables , there is the concept of full refresh and and refresh , if you do the full refresh it will truncate the previous data and only give you the schema , so maybe in that case you have to do the refresh only for those tables in which you dont want to do truncate. check you are doing full refresh or refresh for the tables

Annapurna_Hiriy
New Contributor III
New Contributor III

@zahra Jalilpour​ 

How the DLT tables and views are updated depends on the update type:

  • Refresh all: All live tables are updated to reflect the current state of their input data sources. For all streaming tables, new rows are appended to the table.
  • Full refresh all: All live tables are updated to reflect the current state of their input data sources. For all streaming tables, Delta Live Tables attempts to clear all data from each table and then load all data from the streaming source.

For existing live tables, an update has the same behavior as a SQL REFRESH on a materialized view. For new live tables, the behavior is the same as a SQL CREATE operation.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.