Brahmareddy
Esteemed Contributor II

Hi Dharanip,

From what you’ve described, your logic looks good, but the issue comes from how Materialized Views work in Databricks. They only support incremental updates when the query is very simple and fully predictable (or "deterministic"). Since you're using dynamic filtering with last_timestamp (which changes every run) and reading from a streaming table, Databricks sees the plan as too complex or not supported for incremental refresh.

A better option for your use case would be to use Delta Live Tables (DLT) with streaming. DLT is built to handle incremental data out of the box—you just read your Silver table as a stream, filter the active records, apply your transformations, and DLT takes care of the rest. It’s more flexible than Materialized Views for this kind of logic. 

Regards,

Brahma

View solution in original post