- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-30-2025 08:37 AM
Hi,
I have in my bronze schema json files, I am flattening them in a dataframe after that I am creating materialized views in a dlt pipeline, however, in production is taking a lot of time (over 3 hours) is not even a lot of data the biggest materialized views has only 370k rows.
Any advice? should I use delta tables instead of dlt pipelines? or what do you do?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-30-2025 03:59 PM
isnt materilized view recompute on every run? did you try with incremental processing?
Do you have any complex joins?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-31-2025 11:57 AM
At the end was a query problem, I was using crossjoin I removed this and my dlt pipeline worked well.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-31-2025 01:20 PM
Hello @carlos_tasayco
As I mentioned whether you used any join, just wanted to ask, were this cross join were joining two large tables?
