DLT Serverless incremental refresh of materialized view
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-23-2024 05:53 AM - edited 10-23-2024 05:57 AM
I have a materialized view that always does a "COMPLETE_RECOMPUTE", but I can't figure out why.
I found how I can get the logs:
SELECT * FROM event_log(pipeline_id)
WHERE event_type = 'planning_information'
ORDER BY timestamp desc;
And for my table I got this info:
{
"planning_information": {
"technique_information": [
{
"maintenance_type": "MAINTENANCE_TYPE_GENERIC_AGGREGATE",
"incrementalization_issues": [
{
"issue_type": "PLAN_NOT_INCREMENTALIZABLE",
"prevent_incrementalization": true,
"operator_name": "Aggregate",
"plan_not_incrementalizable_sub_type": "AGGREGATE_NOT_TOP_NODE"
}
]
},
{
"maintenance_type": "MAINTENANCE_TYPE_ROW_BASED",
"incrementalization_issues": [
{
"issue_type": "PLAN_NOT_INCREMENTALIZABLE",
"prevent_incrementalization": true,
"operator_name": "Join",
"join_type": "LEFT_OUTER"
}
]
},
{
"maintenance_type": "MAINTENANCE_TYPE_COMPLETE_RECOMPUTE",
"is_chosen": true,
"is_applicable": true,
"cost": 34936819266
}
],
}
It seems like there is an issue with the left outer join?
But normally that is a supported operation:
https://learn.microsoft.com/en-us/azure/databricks/optimizations/incremental-refresh#enzyme-support