ilarsen
Contributor

I'll offer my opinion.  I see both of those checks (and treatments, if you're converting types for example) as something for the clean/silver/staging/whatever-you-call-it layer.  For us, our bronze layer represents the source data as-is, with SCD type 2 behaviour to keep history.  The only deduplication we're doing in our bronze layer processing is during the structured streaming merge microbatch function.