i have created a materialized view table using delta live table pipeline and its not appending data

[zero234]

i have created a materialized view table using delta live table pipeline , for some reason it is overwriting data every day , i want it to append data to the table instead of doing full refresh suppose i had 8 million records in table and if i

run the pipeline it will remove those previous records and only put in new records. i want it to appends to already existing data i have tried using @Dlt.table(merge Mode="append")it throws unexpected keyword argument error

i have tried using @Dlt.table(merge Mode="append")it throws unexpected keyword argument error

what to do so my pipeline appends data 

[Kaniz · Accepted solution]

Hi @zero234., To ensure that your Delta Live Table pipeline appends data instead of overwriting it, you can use the @append_flow decorator.

Here are the steps:

1. Use @append_flow:

   • The @append_flow decorator is in Public Preview and allows you to write to a streaming table from multiple streaming sources.
   • It enables you to add and remove streaming sources that append data to an existing streaming table w...¹².
   • Apply this decorator to your pipeline to ensure that new data is appended to the existing table.

2. Define Materialized Views or Streaming Tables:

   • In Python, Delta Live Tables determines whether to update a dataset as a materialized view or a streaming table based on the defining query.
   • Use the @table decorator to define both materialized views and streaming tables¹.

3. Example Usage:

   from deltatables import Dlt
   
   @Dlt.table
   def my_materialized_view():
       # Your query here (e.g., SELECT * FROM my_source_data)
       pass
   
   @Dlt.append_flow
   def my_streaming_pipeline():
       # Your streaming logic here
       pass
   

4. Override Default Behavior:

   • By default, streaming tables require append-only data sources.
   • However, if your streaming source is another streaming table that requires updates or deletes, you c...³⁴.
   • This is useful when ingesting source data to create initial datasets in a pipeline.

[kulkpd]

@zero234 ,

Adding some suggestion based on answers from @Kaniz. Important point to note here: "To define a materialized view in Python, apply @table to a query that performs a static read against a data source. To define a streaming table, apply @table to a query that performs a streaming read against a data source."

I think if you which to read by streaming mode, DLT will treat your destination as streaming table.