DLT Gold aggregation with apply_change

deepu1
New Contributor

‎01-12-2026 05:31 AM

I am building a Gold table using Delta Live Tables (DLT). The Gold table contains aggregated data derived from a Silver table. Aggregation happens monthly. However, the requirement is Only the current (year, month) should be recalculated. Previous months must remain unchanged even if there is a change of data silver.

If I use @Dlt.table, the entire aggregation is recomputed every run

I tried using dlt.apply_changes, but I encountered errors as my filtered dataframe doesnt have complete data as it has only the data for the processing month.

 

Below is my apply_changes framework.

def apply_changes_to_table(table_name, compute_fn, apply_changes_keys😞
    table_schema, table_description = load_schema_from_yaml(table_name)
    processed_source_table = f"{table_name}_staging"

    @dlt.table(name=processed_source_table)
    def create_processed_source():
        return compute_fn()  # Already filtered

    dlt.create_streaming_table(
        name=table_name,
        comment=table_description,
        schema=table_schema,
    )

    dlt.create_auto_cdc_flow(
        target=table_name,
        source=processed_source_table,
        keys=apply_changes_keys,
        sequence_by=col("updated_timestamp"),
        stored_as_scd_type=1
    )
 
I want to understand, if Is it possible to do partial (current-month-only) aggregation updates in the Gold layer using apply_changes?

 

0 Kudos