Thank you @Retired_mod. This makes sense. But can you elaborate more on how to use spark's streaming metrics? Where they can be found or are there any articles that explains how it works? Thanks!