aleksandra_ch
Databricks Employee
Databricks Employee
Hi Alf0 and welcome to the Databricks Community!

The Lakeflow Spark Declarative Pipelines (SDP) cost model considers multiple factors when deciding whether to perform an incremental refresh or a full recompute. It makes a best-effort attempt to incrementally refresh results for all supported operations.To address your specific observations:
  • The "Cost" Field: The cost field you see in the event log is a legacy attribute and is not actually utilized by the current cost model. It is slated for removal soon, so you can safely ignore it for now.
  • Manual Control: While the model is generally accurate, we recognize the need for more control. We are working on a feature that will allow users to explicitly define the refresh strategy (incremental vs. full recompute) for Materialized Views (MVs). Stay tuned for updates!
Best Practices for Ensuring Incremental Refresh:If you are seeing unexpected full recomputes, consider these optimizations:
  • Decompose Complex MVs: Split large, complex MVs into multiple smaller ones. Excessive joins or deeply nested operators can sometimes exceed the complexity threshold for incrementalization.
  • Increase Update Frequency: If source tables change significantly between runs, the model may determine a full recompute is cheaper. If you see CHANGESET_SIZE_THRESHOLD_EXCEEDED in your logs, try running updates more frequently to reduce the volume of changes per update.
  • Ensure Deletion Vectors and Row-Level Tracking are enabled on your source tables: Deletion vectors minimize the changeset size, and row-level tracking is a prerequisite for incrementalizing certain operators.
  • Non-Deterministic Functions: These are generally supported in WHERE clauses. Support for operators is constantly updated in the doc: Azure l AWS l GCP. 
For a deeper dive into how these recomputes are calculated, I recommend this technical blog: Optimizing Materialized Views Recomputes.

Hope this helps! 

View solution in original post