- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
4 weeks ago
Hey @alejandro_jaram, You can check if the DLT/Lakeflow event log for the affected materialized view. Compare one fast successful run with one slow/stuck run and check the planning_information details. Specifically, look for whether the MV is using an incremental plan or full recompute and whether there is any reason listed for rejecting incremental refresh such as plan not incrementalizable/row tracking not enabled/the cost model rejecting the incremental plan etc
You can also try to isolate whether one specific materialized view is causing the issue. If possible, refresh or run only the affected MV instead of the whole pipeline. If one MV is taking most of the time, then the problem is likely in that MV’s query plan/source tables/joins/aggregations or metric view materialization path
Since full recompute itself is slow for only 2 GB you should even consider checking these:
1) skew on join or group by keys
2) small files or file churn in the source tables
3) an expensive join strategy
4) row filters, column masks, expectations, or non deterministic expressions
5) metric views referencing other metric views/materialized views
6) source table properties changing in a way that affects incremental refresh
7) row tracking or CDF not enabled on source Delta tables where incremental refresh depends on them
Another useful test is to run the underlying SQL for the problematic MV outside the DLT/Lakeflow pipeline if possible. If the same query runs quickly outside the pipeline but is slow inside the pipeline that points more towards the materialization/runtime path. If it is slow outside as well then the issue is likely in the query plan or source table layout.