Databricks Community

VMeghraj · 09-06-2023

By default SHS uses spark.history.fs.numReplayThreads = 25% of avaliable cores (Number of threads that will be used by history server to process event logs)How can we increase the number of cores for Spark History Server ?

VMeghraj · 08-30-2023

Creating an application to capture cluster metrics and sending HTTP REST request to the Spark History Server's API endpoint to retrieve a list of applications. This request doesn't generate logs in the Spark History Server's log files. The Spark Hist...

VMeghraj · 09-07-2023

Setting the value in % will throw an error (spark.history.fs.numReplayThreads=50%)We need to provide int as an value spark.history.fs.numReplayThreads=25Stack Trace :Exception in thread "main" java.lang.reflect.InvocationTargetExceptionat java.base/j...

VMeghraj · 08-31-2023

Do you have any examples how it can be logged on client side ? Enabling verbose logging on curl request dosen't help much.e.g) curl -v http://<hostname>:18080/api/v1/applications/

Databricks Community

User Stats

User Activity

Increase cores for Spark History Server

REST API

Re: Increase cores for Spark History Server

Re: REST API