nikhilj0421
Databricks Employee
Databricks Employee

Hi @Ranga_naik1180, adding to the above point: 

Since we have enabled the apply changes on the silver layer, it is updating the records successfully coming from the bronze table.Now, in the silver layer, that record is marked as an UPDATE instead of an INSERT, so we need to enable change data feed in the source table(silver layer table) of the gold layer. 

To resolve this, we can go with the below solution:

 To enable it, follow the process below:

  • Enable the change data feed in the silver table:
dlt.create_target_table(

name="silver",

table_properties = {"delta.enableChangeDataFeed": "true"}

)
  • Read change data feed from source table:
@dlt.view

def intermediate_update_view():

  return (

    spark.readStream

    .option("readChangeFeed", "true")

    .table("source_table")

    .filter(col("_change_type").isin(["update_postimage", "insert", "delete"]))

  )

 

View solution in original post