pradeep_singh
Contributor III

I believe you are looking for two things here - 

  1. Incrementally refreshing gold tables(materialized views ) 
  2. History on those tables ( materialized views ) 

You can get the first feature with materialized views in gold layer if you use SDP (Serverless Pipelines - Powered by Enzyme ) . History is still problematic with MVs . This second requirement would needs a regular delta table . Thats when you will have to switch to using Delta CDF (change data feed ) with structured streaming . 



Thank You
Pradeep Singh - https://www.linkedin.com/in/dbxdev