So if I understand correctly, the general recommendation for this type of workload is to leverage the new Spark 4 pipelines with Materialized Views for incremental updates, while being mindful of the constraints around logic changes, full refresh behavior, and downstream recomputation.

In our case, it seems that MV-based incremental refresh is the right direction, but we need to carefully design around:
- Logial changes which can trigger big CDC propagation to downstream MVs,
- Architectural choices to minimize cascading recomputations.

From my side, the core questions raised in this thread are now sufficiently clarified, particularly considering the capabilities introduced with the new Spark 4 pipelines. I will open a separate topic focused specifically on some of the more advanced edge cases and architectural patterns for those use cases.

Thank you for the clarifications!