Increase cores for Spark History Server

[VMeghraj]

By default SHS uses spark.history.fs.numReplayThreads = 25% of avaliable cores (Number of threads that will be used by history server to process event logs)

How can we increase the number of cores for Spark History Server ?

[Kaniz_Fatma]

Hi @VMeghraj, To increase the number of cores for Spark History Server, you can modify the spark.history.fs.numReplayThreads Configuration parameter. 

You can set the desired number of cores by modifying the value of spark.history.fs.numReplayThreads in the Spark configuration file (spark-defaults.conf or spark.conf).

For example, if you want to increase the number of cores to 50% of the available seats, you can set the value to spark.history.fs.numReplayThreads=50%.

After modifying the configuration file, you need to restart the Spark History Server for the changes to take effect.

[VMeghraj]

Setting the value in % will throw an error (spark.history.fs.numReplayThreads=50%)

We need to provide int as an value spark.history.fs.numReplayThreads=25

Stack Trace :

Exception in thread "main" java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:337)
at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
Caused by: java.lang.IllegalArgumentException: spark.history.fs.numReplayThreads should be int, but was 50%
at org.apache.spark.internal.config.ConfigHelpers$.toNumber(ConfigBuilder.scala:35)
at org.apache.spark.internal.config.ConfigBuilder.$anonfun$intConf$1(ConfigBuilder.scala:232)
at org.apache.spark.internal.config.ConfigBuilder.$anonfun$intConf$1$adapted(ConfigBuilder.scala:232)
at scala.Option.map(Option.scala:230)
at org.apache.spark.internal.config.ConfigEntryWithDefaultFunction.readFrom(ConfigEntry.scala:173)
at org.apache.spark.SparkConf.get(SparkConf.scala:261)
at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:105)
at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:90)