One of the more frustrating things when working with materialized views in Databricks was checking whether a view had refreshed incrementally. One way to verify it was by checking the event log, but that required running the pipeline and executing a query.
Finally, we now have a better way to verify whether an incremental refresh will be applied.
By using EXPLAIN CREATE MATERIALIZED VIEW , we can easily verify whether our materialized view is eligible for incremental refresh.
The attached screenshot shows an example of this in action. On the left side, my view definition allows incremental refreshes.
On the right side, I used the non-deterministic function current_timestamp, which prevents incrementalization
In this case, we also get a message with a precise explanation of why incremental refresh is not possible.
PS: Keep in mind that Databricks clarified that eligibility ≠ guaranteed incremental execution when using AUTO refresh policy - the optimizer can still choose full recompute if it’s cheaper (so use incremental refresh policy to enforce it)
