The way to visualize logging of Spark Job graphically

[Junda]

Is there any way to visualize logging such as execution runtime or memory usage of Spark Job graphically just like the image below by utilizing Databricks Partner Connect for free?

Also, I'd appreciate to know any other ways to visualize the logging of Spark Job graphically if they exist even though it requires fees.

[szymon_dybczak]

Hi @Junda ,

Your image is more focus on graphical query plan, in that case you can try below approach:

https://blog.gbrueckl.at/2024/04/visualizing-spark-execution-plans/

If you're more interested in CPU usage/memory consumption you can try grafana plugin for databricks

[Junda]

Hi, @szymon_dybczak 

Thank you very much for reply and sharing the article. That was very close to what I had in mind.

[szymon_dybczak]

Pleasure to help 🙂