Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-23-2024 01:18 PM
I'll offer my opinion. I see both of those checks (and treatments, if you're converting types for example) as something for the clean/silver/staging/whatever-you-call-it layer. For us, our bronze layer represents the source data as-is, with SCD type 2 behaviour to keep history. The only deduplication we're doing in our bronze layer processing is during the structured streaming merge microbatch function.