Hi, I have following streaming setup:
I want to remove duplicates in streaming.
1) deduplication strategy is defined by two fields: extraction_timestamp and hash (row wise hash)
2) watermark strategy: extraction_timestamp with "10 seconds" interval
--> Removing duplicates in side of extraction_timestamp
Now rocksdb and state management works fine (it uses watermark correctly and does not need/check old active states --> just I wanted, because I know that there is no need to check these states because extraction time between these streaming batch are more than 10 seconds). The problem is that because extraction_timestamp is part of the deduplication strategy all state are "active" and rocksdb is not cleaning up these old state files.
I wondered is there any other option that "manually" delete these state files to manage storage size (like I say these are not used because of watermark logic)? Why rocksdb is not using watermark as a part of cleaning plan?
