topic Re: change data feed on delta live tables in Data Engineering

change data feed on delta live tables

PriyaAnanthram — Tue, 02 May 2023 14:05:21 GMT

I have a delta live table where I am reading cdc data and merging this data in silver using apply changes. In silver can I find out what all data has changed since the last run similar to change data feed table_changes?

Re: change data feed on delta live tables

-werners- — Wed, 03 May 2023 09:07:25 GMT

if the silver table is a delta lake table, and change data feed is enabled that should be possible.

Re: change data feed on delta live tables

PriyaAnanthram — Wed, 03 May 2023 10:05:10 GMT

thanks for your reply @Werner Stinckens i have enabled change data feed but the issue is every time i run the dlt pipeline an overwrite happens so when i do table_changes between versions i see all the changes as inserts?

Re: change data feed on delta live tables

PriyaAnanthram — Wed, 03 May 2023 10:07:24 GMT

I also have a requirment where i write to a live table (materialized view) and have cdf enabled i want to see the changes but here to i see overwrites happening after dlt pipeline runs

Re: change data feed on delta live tables

-werners- — Wed, 03 May 2023 11:16:50 GMT

CDF should also deliver updates and deletes, not only inserts. Perhaps the way you merge the table?

Re: change data feed on delta live tables

PriyaAnanthram — Wed, 03 May 2023 12:15:48 GMT

I think my main problem is that i havent been able to enable change data feed on the silver layer since its a view..

dlt.create_target_table(f"silver_{schemaName}_{tableName}",table_properties = {'delta.enableChangeDataFeed': 'true'})

I can see the changes so scd is happening...

have been able to enable cdf on the bronze

Error in SQL statement: DeltaAnalysisException: Cannot describe the history of a view.

Re: change data feed on delta live tables

-werners- — Wed, 03 May 2023 12:45:13 GMT

The docs only mention tables, so that probably means views are not supported.