- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-23-2025 07:23 AM - edited 10-23-2025 07:25 AM
Your solution #1 is very frustrating to me as well, for a number of reasons. Simply put, we have to be able to compare incoming data to target data for normal ETL operations.
One way around this is to create a view of your target silver table, outside of your pipeline (this part is key), and compare against that view. The pipeline will give you a warning telling you that you're being naughty, but it will let you do the comparison anyways. We have had to do this and it does work, although it's clearly not a desirable solution to have to maintain this set of views, but it would allow you to perform the deletion check.
Example: `create or replace view silver.vw_target_table as select * from silver.target_table;`
Someone else might be able to chime in with a more elegant solution, but that's what I've got for you at least!