How to blacklist an executor on the Spark cluster

[Hubert-Dudek]

This is a lit of configuration keys to enable or alter the blacklist mechanism:

spark.blacklist.enabled – set to True

spark.blacklist.task.maxTaskAttemptsPerExecutor (1 by default)

spark.blacklist.task.maxTaskAttemptsPerNode (2 by default)

spark.blacklist.application.maxFailedTasksPerExecutor (2 by default)

spark.blacklist.stage.maxFailedTasksPerExecutor (2 by default)

spark.blacklist.application.maxFailedExecutorsPerNode (2 by default)

spark.blacklist.stage.maxFailedExecutorsPerNode (2 by default)

spark.blacklist.timeout (not set by default)

spark.blacklist.killBlacklistedExecutors (false by default)