TPSteve
New Contributor II

First, you need to understand why your current solution is failing.

Materialized views and views in DLT don't differ conceptually from mat. views and views in PostgreSQL. Every time the pipeline is run, both the mat. view and the view will be recalculated. The difference is that a mat. view is persisted while a view is only available during the lifetime of the pipeline. If your backfilled records are not in the mat. views source table, then they will not be included in the refreshed result. You also don't want to recalculate all of the historic data unless you're updating the results by aggregating the historic and new data together. It doesn't sound like that's the use case. You're interested in appending the incremental results to the table.

As always, there are multiple ways to construct a pipeline. Without knowing more details about your data, DLT should be an option. Whether or not it's the most optimal is something else to consider.

I also suggest you follow the lakehouse paradigm. Land the untransformed data in a delta table first and then process the transformations. If you ever need to recalculate all the data, you won't need to read it from PostgreSQL.