By default, the state data (streaming aggregation query) is maintained in the JVM memory of the executors and large number of state objects could put memory pressure on the JVM causing high GC pauses. If you have stateful operations in your streaming query, it is recommended to use a more optimized state management solution based on RocksDB.

More details at https://docs.databricks.com/spark/latest/structured-streaming/production.html#optimize-performance-o...