flattening json in dlt pipeline

[carlos_tasayco]

Hi,

I have in my bronze schema json files, I am flattening them in a dataframe after that I am creating materialized views in a dlt pipeline, however, in production is taking a lot of time (over 3 hours) is not even a lot of data the biggest materialized views has only 370k rows. 

Any advice? should I use delta tables instead of dlt pipelines? or what do you do?

[Khaja_Zaffer]

isnt materilized view recompute on every run? did you try with incremental processing? 

Do you have any complex joins?

[carlos_tasayco · Accepted solution]

At the end was a query problem, I was using crossjoin I removed this and my dlt pipeline worked well.

[Khaja_Zaffer]

Hello @carlos_tasayco 

As I mentioned whether you used any join, just wanted to ask, were this cross join were joining two large tables?