cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

DLT || Python || Aggregate Functions recomputing all the records

shivam-singh
New Contributor

Hi all, 

I am building a realtime dashboard using Databricks Delta Live Tables Pipeline and using the following steps : - 

Bronze Table : Using the autoloader functionality provided by databricks, its incrementally ingesting new files records into a bronze table.
Silver Table : Using the read_stream function provided in spark for structured streaming, we are creating the silver table by filtering the records and selecting few fields from the bronze table that are required.
Gold Table : Using the read function provided in spark for reading complete record, we are creating the gold table, which is the materialized view and also using aggregate function (SUM), and group by clause to create it.


Problem : 
Bronze and silver table are doing incremental ingestion, however incase of gold table, the entire record in the table is getting recomputed everytime a new record is received in the silver table.

What I want to ensure is that for the particular group by clause only updates should be performed and rest of the records are locked and dont require any update. 

I have also tried using streaming table instead of materialized view for gold as well, in this case also the entire records are getting recomputed.
Any help would be appreciated.

1 REPLY 1

Kaniz_Fatma
Community Manager
Community Manager

Hi @shivam-singh

I hope this helps you with your project. You can check out the links I provided if you need more information. Have a great day! 😊

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group