Hi,

To your question, the dataset is large and will be growing as well. In my use case the Silver layer contains a USER table(streaming Delta live table) like below:

USER table:

In the Gold Layer we need a table like:

The transformations that has to be done in Gold Layer are:

1. Renaming of columns to the dataset 

2. To add the PK/FK constraints to the dataset

3. To remove few columns like batchid in the dataset

4. To make the dataset SCD TYPE 1 in the Gold layer i.e to bring only the latest updated records of the user 

5. Sometimes to make join with 1 or 2 tables and bring new columns

The question here is:

Should the dataset be created as a streaming Table or a materialized view?