osingh
Contributor

This is a really interesting find, and honestly not something most people expect from materialized views.

Under the hood, MVs in Databricks declarative pipelines are still Delta tables. So when you set partitionOverwriteMode=dynamic and partition by a column like a date, Delta just does what it’s supposed to do. It overwrites only the partitions present in the write, keeps the rest, and inserts new ones if they don’t exist.

That’s why your results make sense:

-Rows for the same date get replaced
-New rows show up correctly
-Other dates are untouched

Most teams don’t use MVs this way because people usually think of them as “derived and recomputed,” not something that behaves like a controlled overwrite table. Since replaceWhere isn’t allowed, folks normally jump straight to MERGE or streaming tables.

But in cases like yours, where there’s no real primary key and a clear overwrite boundary like a date, this feels like a very reasonable solution. It’s simpler than MERGE and still gives you idempotent behavior.

The big thing to be aware of, like you already mentioned, is full refresh. That would blow everything away and rebuild it. But that’s not really unique to MVs; streaming tables in declarative pipelines have the same risk.

So yeah, it’s not a common pattern, but it’s not wrong either. I’ve seen similar approaches used in batch-style DLT pipelines when people really understand how Delta behaves. As long as it’s documented and intentional, this seems like a valid approach.

Om Singh

View solution in original post