cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

change data feed on delta live tables

PriyaAnanthram
Contributor III

I have a delta live table where I am reading cdc data and merging this data in silver using apply changes. In silver can I find out what all data has changed since the last run similar to change data feed table_changes?

1 ACCEPTED SOLUTION

Accepted Solutions

-werners-
Esteemed Contributor III

The docs only mention tables, so that probably means views are not supported.

View solution in original post

6 REPLIES 6

-werners-
Esteemed Contributor III

if the silver table is a delta lake table, and change data feed is enabled that should be possible.

PriyaAnanthram
Contributor III

thanks for your reply @Werner Stinckens​  i have enabled change data feed but the issue is every time i run the dlt pipeline an overwrite happens so when i do table_changes between versions i see all the changes as inserts?

-werners-
Esteemed Contributor III

CDF should also deliver updates and deletes, not only inserts. Perhaps the way you merge the table?

I think my main problem is that i havent been able to enable change data feed on the silver layer since its a view..

dlt.create_target_table(f"silver_{schemaName}_{tableName}",table_properties = {'delta.enableChangeDataFeed': 'true'})

I can see the changes so scd is happening...

 have been able to enable cdf on the bronze

Error in SQL statement: DeltaAnalysisException: Cannot describe the history of a view.

-werners-
Esteemed Contributor III

The docs only mention tables, so that probably means views are not supported.

PriyaAnanthram
Contributor III

I also have a requirment where i write to a live table (materialized view) and have cdf enabled i want to see the changes but here to i see overwrites happening after dlt pipeline runs

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!