https://community.databricks.com/t5/machine-learning/can-i-get-detailed-metrics-of-the-rocksdb-that-i-have-used-in/m-p/17795#M969 Mon, 28 Jun 2021 13:37:59 GMT User16826994223 2021-06-28T13:37:59Z https://community.databricks.com/t5/machine-learning/can-i-get-detailed-metrics-of-the-rocksdb-that-i-have-used-in/m-p/17795#M969 Mon, 28 Jun 2021 13:37:59 GMT https://community.databricks.com/t5/machine-learning/can-i-get-detailed-metrics-of-the-rocksdb-that-i-have-used-in/m-p/17795#M969 User16826994223 2021-06-28T13:37:59Z https://community.databricks.com/t5/machine-learning/can-i-get-detailed-metrics-of-the-rocksdb-that-i-have-used-in/m-p/17796#M970 <P>If you have configured your Structured Streaming query to use&nbsp;<A href="https://docs.microsoft.com/en-gb/azure/databricks/spark/latest/structured-streaming/production#optimize-performance-of-stateful-streaming-queries" alt="https://docs.microsoft.com/en-gb/azure/databricks/spark/latest/structured-streaming/production#optimize-performance-of-stateful-streaming-queries" target="_blank">RocksDB as the state store</A>, you can now get better visibility into the performance of RocksDB, with detailed metrics on get/put latencies, compaction latencies, cache hits, and so on. These metrics are available through the&nbsp;</P><P>StreamingQueryProgress and&nbsp;StreamingQueryListener</P><P>&nbsp;APIs for&nbsp;<A href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#monitoring-streaming-queries" alt="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#monitoring-streaming-queries" target="_blank">monitoring a streaming query</A>.</P><P></P><P>More Info -https://docs.microsoft.com/en-gb/azure/databricks/spark/latest/structured-streaming/production#rocksdb-metrics</P> Mon, 28 Jun 2021 13:38:51 GMT https://community.databricks.com/t5/machine-learning/can-i-get-detailed-metrics-of-the-rocksdb-that-i-have-used-in/m-p/17796#M970 User16826994223 2021-06-28T13:38:51Z