Using DLT with a non-streaming large table

[pskchai]

We have a source table that receives daily append operations, but the rows created within the last 30 days in this table can be updated or deleted. Thus, the source table is not exactly a streaming source.

Our processing workflow involves performing "GROUP BY" and aggregations on this source table. Due to the table's large size, exceeding 1TB, recomputing the entire table for updating downstream tables would incur significant computation costs.

My question is:

Do materialized views get updated when the underlying source table changes, without requiring a recomputation of the entire source table? If not, I would greatly appreciate your guidance on the best methods to handle this problem efficiently.

[Chengcheng · Accepted solution]

If your source table is a delta table or cloud file where DLT can monitor any changes.

The answer would YES I think.

That is exactly what DLT Materialized view does.

See documentation of DLT for details:

https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/#materialized-view

[Anonymous]

Hi @Pongsakorn Chairatanakul​ 

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you. 

Cheers!