SCD type2 snowflake

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.

I have monthly files which comes in S3 bucket. I want to implement SCD type2 in snowflake.

I am ok to read the new files, clean it.

My question is about comparing what I have read from the files, with what is stored in the snowflake table already(millions of records) and capturing the change.

Main question:

How to join the snowflake table to the view created from the monthly file.

1)I can read the snowflake table in data bricks like select * and can create a view and do the processing.

I am sure this is now how it's done in production.

Thanks,

Any pointers on how to solve the problem?