https://community.databricks.com/t5/data-engineering/scd-type2-snowflake/m-p/32549#M23724 <P>I have monthly files which comes in S3 bucket. I want to implement SCD type2 in snowflake.</P><P></P><P>I am ok to read the new files, clean it.</P><P></P><P>My question is about comparing what I have read from the files, with what is stored in the snowflake table already(millions of records) and capturing the change.</P><P></P><P>Main question:</P><P>How to join the snowflake table to the view created from the monthly file.</P><P></P><P>1)I can read the snowflake table in data bricks like select * and can create a view and do the processing.</P><P>I am sure this is now how it's done in production.</P><P></P><P>Thanks,</P><P></P><P>Any pointers on how to solve the problem?</P><P></P><P></P><P></P><P></P><P></P><P></P><P></P><P></P><P></P> Mon, 05 Sep 2022 10:03:10 GMT sabooalex 2022-09-05T10:03:10Z https://community.databricks.com/t5/data-engineering/scd-type2-snowflake/m-p/32549#M23724 <P>I have monthly files which comes in S3 bucket. I want to implement SCD type2 in snowflake.</P><P></P><P>I am ok to read the new files, clean it.</P><P></P><P>My question is about comparing what I have read from the files, with what is stored in the snowflake table already(millions of records) and capturing the change.</P><P></P><P>Main question:</P><P>How to join the snowflake table to the view created from the monthly file.</P><P></P><P>1)I can read the snowflake table in data bricks like select * and can create a view and do the processing.</P><P>I am sure this is now how it's done in production.</P><P></P><P>Thanks,</P><P></P><P>Any pointers on how to solve the problem?</P><P></P><P></P><P></P><P></P><P></P><P></P><P></P><P></P><P></P> Mon, 05 Sep 2022 10:03:10 GMT https://community.databricks.com/t5/data-engineering/scd-type2-snowflake/m-p/32549#M23724 sabooalex 2022-09-05T10:03:10Z