Andolina
New Contributor III

Thank you both for replying. We are indeed using the natural keys from source for auto-cdc keys. The Identity column is not used in auto-cdc but needs to be populated as an added column in my silver tables. Users need that to join multiple tables in gold layer. Now the problem I am facing is with auto-cdc I am not able to populate that directly in silver as there is no such option. 

However, I found an alternative way where we create the tables upfront in a separate script. Looks like I cannot do this in python as dp.create_streaming_table() didn't support in my case (may be there is some way here too), but I am able to achieve it using a .sql script. The .sql script needs to be part of the declarative pipeline though.

Example:

CREATE OR REPLACE STREAMING TABLE <tablename>
(
key_column BIGINT GENERATED BY DEFAULT AS IDENTITY (START WITH 1),
..
)

The issue I have now is I am not aware of any potential risk of doing this yet. Any insight would be helpful.
@aleksandra_ch - Thank you for the "Track History On" option. Let me check on that. I only populate that column while generating silver. Do you think I can set this option on the identity column?


Thanks,
Andolina