Download event and run logs

[hpicatto]

how can I download the run and event logs? spark UI is loading them from somewhere but I couldn't find them in dbfs nor on s3

[Kaniz]

Hi @hpicatto, 

• The Spark Web UI provides a suite of user interfaces (UIs) for monitoring your Spark cluster. You can access it by navigating to http://<driver-node-ip>:18080 in your web browser.
• On the appropriate application page, click the Download button to obtain the logs.
• Unzip the downloaded files and store them in a separate directory (e.g., /Users/imran/Desktop/spark-...¹.
• If you have Spark’s history server running and the application’s event logs exist, you can construct the UI through the history server.
• Start the history server by executing ./sbin/start-history-server.sh.
• The history server creates a web interface at http://<server-url>:18080 (by default), listing both i...
• To enable event logs, add the following configuration to conf/spark-default.conf:

  spark.eventLog.enabled true
  spark.eventLog.dir file:///Users/rockieyang/git/spark/spark-events
  

• Adjust the spark.eventLog.dir path to your desired location