Why do I always see "Executor heartbeat timed out" messages in the Spark Driver logs

[brickster_2018]

Often, I see "Executor heartbeat timed out" messages in the Spark driver logs. Sometimes job fails with this error.

Will increasing "spark.executor.heartbeatInterval" help to mitigate the issue ?

[brickster_2018 · Accepted solution]

This is a common misconception that increasing "spark.executor.heartbeatInterval" will help to mitigate or resolve the heartbeat issues. In fact, increasing the spark.executor.heartbeatInterval will increase the chance of the error and worse the situation.

This is because "spark.executor.heartbeatInterval" determines the interval in which the heartbeat has to be sent. Increasing it will reduce the number of heart beats sent and when the Spark driver checks for the heartbeat every 2 minutes, there is more chance for failure.

To mitigate the issue "spark.network.timeout" can be increased. May to 300 s. Setting a very high value for spark.network.timeout is not recommended as that would delay the true failures.