Hey @Avnish_Jain, we're implementing a very similar inter and intra-batch deduplication process albeit with SCD type-1. However,  we are afraid the drop_duplicates() (in your case dropDuplicates(["data_hash"])) might be looking at the whole stream of data as it could be stateful. How do we ensure, the drop_duplicates is only looking at the data inside the microbatch?