https://community.databricks.com/t5/get-started-discussions/increase-cores-for-spark-history-server/m-p/43848#M5786 <P>By default SHS uses spark.history.fs.numReplayThreads = 25% of avaliable cores (<SPAN>Number of threads that will be used by history server to process event logs</SPAN>)</P><P>How can we increase the number of cores for Spark History Server ?</P> Wed, 06 Sep 2023 16:34:03 GMT VMeghraj 2023-09-06T16:34:03Z https://community.databricks.com/t5/get-started-discussions/increase-cores-for-spark-history-server/m-p/43848#M5786 <P>By default SHS uses spark.history.fs.numReplayThreads = 25% of avaliable cores (<SPAN>Number of threads that will be used by history server to process event logs</SPAN>)</P><P>How can we increase the number of cores for Spark History Server ?</P> Wed, 06 Sep 2023 16:34:03 GMT https://community.databricks.com/t5/get-started-discussions/increase-cores-for-spark-history-server/m-p/43848#M5786 VMeghraj 2023-09-06T16:34:03Z https://community.databricks.com/t5/get-started-discussions/increase-cores-for-spark-history-server/m-p/44008#M5788 <P>Setting the value in % will throw an error (<SPAN>spark.history.fs.numReplayThreads=<STRONG>50%</STRONG>)</SPAN></P><P><SPAN>We need to provide <STRONG>int</STRONG> as an value&nbsp;spark.history.fs.numReplayThreads=<STRONG>25</STRONG></SPAN></P><P>Stack Trace :</P><P>Exception in thread "main" java.lang.reflect.InvocationTargetException<BR />at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)<BR />at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)<BR />at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)<BR />at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)<BR />at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:337)<BR />at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)<BR />Caused by: java.lang.IllegalArgumentException: spark.history.fs.numReplayThreads should be int, but was 50%<BR />at org.apache.spark.internal.config.ConfigHelpers$.toNumber(ConfigBuilder.scala:35)<BR />at org.apache.spark.internal.config.ConfigBuilder.$anonfun$intConf$1(ConfigBuilder.scala:232)<BR />at org.apache.spark.internal.config.ConfigBuilder.$anonfun$intConf$1$adapted(ConfigBuilder.scala:232)<BR />at scala.Option.map(Option.scala:230)<BR />at org.apache.spark.internal.config.ConfigEntryWithDefaultFunction.readFrom(ConfigEntry.scala:173)<BR />at org.apache.spark.SparkConf.get(SparkConf.scala:261)<BR />at org.apache.spark.deploy.history.FsHistoryProvider.&lt;init&gt;(FsHistoryProvider.scala:105)<BR />at org.apache.spark.deploy.history.FsHistoryProvider.&lt;init&gt;(FsHistoryProvider.scala:90)</P> Thu, 07 Sep 2023 15:15:55 GMT https://community.databricks.com/t5/get-started-discussions/increase-cores-for-spark-history-server/m-p/44008#M5788 VMeghraj 2023-09-07T15:15:55Z