Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-21-2026 01:10 PM
Ok, I see - so with the updated scenario above, the easiest option would be to add a second streaming writer for df3a to the same existing f"{silver_catalog}.{silver_schema}.my_silver_table". I would only stick with the easiest option if this is an append-only table and if you are not trying to handle duplicates here.
If this could introduce duplicates, than the first thing you posted is more recommended, where you handle the duplicates by doing a union with all 3 dataframes. One thing I didn't consider on my first response is that you can use "withWatermark" + "dropDuplicatesWithinWatermark" instead of just dropDuplicates