topic Re: The way to visualize logging of Spark Job graphically in Get Started Discussions

The way to visualize logging of Spark Job graphically

Junda — Sat, 20 Jul 2024 02:59:05 GMT

Is there any way to visualize logging such as execution runtime or memory usage of Spark Job graphically just like the image below by utilizing Databricks Partner Connect for free?

Also, I'd appreciate to know any other ways to visualize the logging of Spark Job graphically if they exist even though it requires fees.

Re: The way to visualize logging of Spark Job graphically

szymon_dybczak — Sat, 20 Jul 2024 03:36:17 GMT

Hi @Junda ,

Your image is more focus on graphical query plan, in that case you can try below approach:

https://blog.gbrueckl.at/2024/04/visualizing-spark-execution-plans/

If you're more interested in CPU usage/memory consumption you can try grafana plugin for databricks

Re: The way to visualize logging of Spark Job graphically

Junda — Sat, 20 Jul 2024 10:38:35 GMT

Hi, @szymon_dybczak

Thank you very much for reply and sharing the article. That was very close to what I had in mind.

Re: The way to visualize logging of Spark Job graphically

szymon_dybczak — Sat, 20 Jul 2024 10:58:21 GMT

Pleasure to help 🙂