- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-10-2023 02:17 AM
Hi @Kaniz Fatma ,
I am working on a use case where I am keeping customer data using medallion architecture using Delta live Tables.
But I would like to also delete data based on GDPR. So, I have tried deleting using simple delete script basically deleting that consumer older than 5 years from bronze, silver and gold tables.
After that, I tried to run DLT pipeline again and ran into issue like mentioned above.
" Detected a data update (for example part-00000-6723832a-b8ca-4a20-b576-d69bd5e42652-c000.snappy.parquet) in the source table at version 11. This is currently not supported. If you'd like to ignore updates, set the option 'ignoreChanges' to 'true'. If you would like the data update to be reflected, please restart this query with a fresh checkpoint directory."
Any idea how to implement ignore changes and ignore deletes in DLT?