-
Limit Pipeline Parallelism: Modify the pipeline's configuration to reduce the maximum concurrency for DLT task execution, forcing more serialized or grouped updates.
-
Restructure Pipeline Graph: Instead of 300+ separate materialized views, consider batching tables with no new data into fewer logical processing stages or introducing dummy dependencies to force sequential operation.
-
DLT Table Options: For tables that frequently have no new data, use DLT or Delta Lake configuration options to skip "empty commits" or checkpoint updates unless new data is present, where available.
-
Databricks Contact: If the issue persists after tuning concurrency, coordinate with Databricks support to review orchestration logs for deeper root cause analysis, as it may uncover a bug or undocumented behavior in Lakeflow for high-output-count pipelines.
