@Suteja Kanuri​ - Thank you for this. Been pulled onto other projects and getting back to this now, so will try your solution.

I think the de-duplication option is the one we want, but I'm not certain I'm understanding fully.

I have a call to an api that retrieves a json file each night. As this data changes infrequently, the records are normally identical.

If I understand your de-duped_stream correctly, this will select the latest version of the json file.

Won't this just change each night when I retrieve a new json file, therefore creating a new duplicate entry in the SCD at that point?

I think I'm missing how the de-duplication comes into effect, which is probably due to my lack of experience in this area.

Thank you for your help.