Hi all,

I am a junior engineer and I am working on a use case involving DLT pipelines that read from a csv source at regular intervals and populate tables based upon processing of this. In most cases, I'd like the table contents to be overwritten with the new data, however there is one table where I need to append the new data instead to create a historical view.

To put the situation in pseudocode:

import pipelines as dp

dp.table(name = "Table1") # Overwrite
def table1():
    return spark.read.csv(path_to_my_csv).do_some_simple_processing()

dp.table(name = "Table2") # Need to have this append rather than overwrite
def table2():
    return spark.read.table("Table1")

I've looked and I don't think this is covered by any of the option flags in the table decorator. I've also seen streaming_table and append_flow as potentially recommended, however they seem to require the source itself to be streaming tables which I don't know if this is possible for my use case. Can I ask what the recommended way to achieve this outcome with declarative pipelines would be?

Thanks!