- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-02-2023 07:05 AM
I have a delta live table where I am reading cdc data and merging this data in silver using apply changes. In silver can I find out what all data has changed since the last run similar to change data feed table_changes?
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-03-2023 05:45 AM
The docs only mention tables, so that probably means views are not supported.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-03-2023 02:07 AM
if the silver table is a delta lake table, and change data feed is enabled that should be possible.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-03-2023 03:05 AM
thanks for your reply @Werner Stinckens i have enabled change data feed but the issue is every time i run the dlt pipeline an overwrite happens so when i do table_changes between versions i see all the changes as inserts?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-03-2023 04:16 AM
CDF should also deliver updates and deletes, not only inserts. Perhaps the way you merge the table?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-03-2023 05:15 AM
I think my main problem is that i havent been able to enable change data feed on the silver layer since its a view..
dlt.create_target_table(f"silver_{schemaName}_{tableName}",table_properties = {'delta.enableChangeDataFeed': 'true'})
I can see the changes so scd is happening...
have been able to enable cdf on the bronze
Error in SQL statement: DeltaAnalysisException: Cannot describe the history of a view.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-03-2023 05:45 AM
The docs only mention tables, so that probably means views are not supported.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-03-2023 03:07 AM
I also have a requirment where i write to a live table (materialized view) and have cdf enabled i want to see the changes but here to i see overwrites happening after dlt pipeline runs

