Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
a month ago
I have a Data Lake Transformation (DLT) pipeline that runs weekly. Normally, it takes 8 minutes to complete, but since last Friday (June 19), it has been running for hours until it encounters an out-of-memory error. This pipeline is responsible for creating materialized views from metric views. The only idea I have so far is that the issue might be related to reading materialized views of those metric views. When I perform a full refresh, the materializations of the metric views are refreshed, and the entire pipeline completes in 20 minutes. However, this option is not feasible for production environments due to its cost.