Deduplication with rocksdb, should old state files be deleted manually (to manage storage size)?

LasseL — Fri, 14 Mar 2025 10:59:11 GMT

Hi, I have following streaming setup:

I want to remove duplicates in streaming.

1) deduplication strategy is defined by two fields: extraction_timestamp and hash (row wise hash)

2) watermark strategy: extraction_timestamp with "10 seconds" interval

--> Removing duplicates in side of extraction_timestamp

Now rocksdb and state management works fine (it uses watermark correctly and does not need/check old active states --> just I wanted, because I know that there is no need to check these states because extraction time between these streaming batch are more than 10 seconds). The problem is that because extraction_timestamp is part of the deduplication strategy all state are "active" and rocksdb is not cleaning up these old state files.

I wondered is there any other option that "manually" delete these state files to manage storage size (like I say these are not used because of watermark logic)? Why rocksdb is not using watermark as a part of cleaning plan?

Re: Deduplication with rocksdb, should old state files be deleted manually (to manage storage size)?

LasseL — Mon, 17 Mar 2025 16:34:21 GMT

Found solution. https://kb.databricks.com/streaming/how-to-efficiently-manage-state-store-files-in-apache-spark-streaming-applications <-- these two parameters.

topic Deduplication with rocksdb, should old state files be deleted manually (to manage storage size)? in Get Started Discussions

Deduplication with rocksdb, should old state files be deleted manually (to manage storage size)?

Re: Deduplication with rocksdb, should old state files be deleted manually (to manage storage size)?