Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-14-2026 12:27 AM
Hi Alf0 and welcome to the Databricks Community!
The Lakeflow Spark Declarative Pipelines (SDP) cost model considers multiple factors when deciding whether to perform an incremental refresh or a full recompute. It makes a best-effort attempt to incrementally refresh results for all supported operations.To address your specific observations:
The Lakeflow Spark Declarative Pipelines (SDP) cost model considers multiple factors when deciding whether to perform an incremental refresh or a full recompute. It makes a best-effort attempt to incrementally refresh results for all supported operations.To address your specific observations:
- The "Cost" Field: The cost field you see in the event log is a legacy attribute and is not actually utilized by the current cost model. It is slated for removal soon, so you can safely ignore it for now.
- Manual Control: While the model is generally accurate, we recognize the need for more control. We are working on a feature that will allow users to explicitly define the refresh strategy (incremental vs. full recompute) for Materialized Views (MVs). Stay tuned for updates!
Best Practices for Ensuring Incremental Refresh:If you are seeing unexpected full recomputes, consider these optimizations:
- Decompose Complex MVs: Split large, complex MVs into multiple smaller ones. Excessive joins or deeply nested operators can sometimes exceed the complexity threshold for incrementalization.
- Increase Update Frequency: If source tables change significantly between runs, the model may determine a full recompute is cheaper. If you see
CHANGESET_SIZE_THRESHOLD_EXCEEDEDin your logs, try running updates more frequently to reduce the volume of changes per update. - Ensure Deletion Vectors and Row-Level Tracking are enabled on your source tables: Deletion vectors minimize the changeset size, and row-level tracking is a prerequisite for incrementalizing certain operators.
- Non-Deterministic Functions: These are generally supported in
WHEREclauses. Support for operators is constantly updated in the doc: Azure l AWS l GCP.
For a deeper dive into how these recomputes are calculated, I recommend this technical blog: Optimizing Materialized Views Recomputes.
Hope this helps!
Hope this helps!