topic Re: flattening json in dlt pipeline in Data Engineering

flattening json in dlt pipeline

carlos_tasayco — Wed, 30 Jul 2025 15:37:43 GMT

Hi,

I have in my bronze schema json files, I am flattening them in a dataframe after that I am creating materialized views in a dlt pipeline, however, in production is taking a lot of time (over 3 hours) is not even a lot of data the biggest materialized views has only 370k rows.

Any advice? should I use delta tables instead of dlt pipelines? or what do you do?

Re: flattening json in dlt pipeline

Khaja_Zaffer — Wed, 30 Jul 2025 22:59:27 GMT

isnt materilized view recompute on every run? did you try with incremental processing?

Do you have any complex joins?

Re: flattening json in dlt pipeline

carlos_tasayco — Thu, 31 Jul 2025 18:57:35 GMT

At the end was a query problem, I was using crossjoin I removed this and my dlt pipeline worked well.

Re: flattening json in dlt pipeline

Khaja_Zaffer — Thu, 31 Jul 2025 20:20:04 GMT

Hello @carlos_tasayco

As I mentioned whether you used any join, just wanted to ask, were this cross join were joining two large tables?