Simple append only in DLT

andreacfm — Thu, 04 Dec 2025 13:45:07 GMT

I am facing an issue trying to find a way to insert some computed rows into a table in the context of a dlt pipeline.

My use case is extremely simple. Moving from bronze to silver I update several tables using a mix of streaming and materialized tables. I need also to write some rows into a silver table. I do not have a stream source to read as the rows are computed/aggregated. I tried the append_flow but that expects a stream source that I cannot provide. Using once=true works but as the param says it works only once.

It seems extremely weird that DB does not support a so simple use case. Am I missing something?

Re: Simple append only in DLT

Louis_Frolio — Thu, 04 Dec 2025 14:31:13 GMT

Greetings @andreacfm,

You’re not missing a thing. What you’re seeing is a known limitation in how DLT/Lakeflow pipelines handle append_flow. It really does expect a streaming source, and the once=True flag only fires during the first run of the pipeline or when you explicitly ask for a full refresh. So your observations line up with the intended behavior.

Let’s dig into the practical workarounds that teams typically use when they need to insert computed rows without a true streaming source.

Workarounds for inserting computed rows

Materialized views for aggregations

For silver-layer rollups or derived metrics, materialized views are the cleanest pattern. They’re designed for batch-style transforms, and you get predictable refresh behavior without pretending your data came from a stream.

Example:

@dlt.table(name="silver.aggregated_metrics")
def aggregated_metrics():
    return (
        dlt.read("bronze.source_table")
            .groupBy("dimension")
            .agg(sum("metric").alias("total"))
    )

Static-lookup table pattern

If you’re dealing with reference data — static rows, small business logic tables, etc. — define the table outside of a streaming context. DLT will treat it as a materialized view, and you can join it with streaming tables using stream-to-static joins. This preserves lineage and avoids bending the streaming model in ways it doesn’t support.

One-time batch append with once=True

If your logic truly needs to run only during backfills or initial ingestions, then yes, append_flow with once=True is the intended pattern. But it won’t fire repeatedly — which is exactly what you’ve run into.

Why the limitation exists

DLT is deliberately built around declarative, incremental processing. It expects data to come from sources it can track over time. Arbitrary row insertion without a traceable upstream source breaks that model — especially around incremental refresh, lineage, and reproducibility. So the framework simply doesn’t allow it unless you wrap the logic in a source-driven or MV-driven construct.

Hope this helps clarify the landscape.

Regards, Louis.