cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Why do I always see "Executor heartbeat timed out" messages in the Spark Driver logs

User16869510359
Esteemed Contributor

Often, I see "Executor heartbeat timed out" messages in the Spark driver logs. Sometimes job fails with this error.

Will increasing "spark.executor.heartbeatInterval" help to mitigate the issue ?

1 ACCEPTED SOLUTION

Accepted Solutions

User16869510359
Esteemed Contributor

This is a common misconception that increasing "spark.executor.heartbeatInterval" will help to mitigate or resolve the heartbeat issues. In fact, increasing the spark.executor.heartbeatInterval will increase the chance of the error and worse the situation.

This is because "spark.executor.heartbeatInterval" determines the interval in which the heartbeat has to be sent. Increasing it will reduce the number of heart beats sent and when the Spark driver checks for the heartbeat every 2 minutes, there is more chance for failure.

To mitigate the issue "spark.network.timeout" can be increased. May to 300 s. Setting a very high value for spark.network.timeout is not recommended as that would delay the true failures.

View solution in original post

1 REPLY 1

User16869510359
Esteemed Contributor

This is a common misconception that increasing "spark.executor.heartbeatInterval" will help to mitigate or resolve the heartbeat issues. In fact, increasing the spark.executor.heartbeatInterval will increase the chance of the error and worse the situation.

This is because "spark.executor.heartbeatInterval" determines the interval in which the heartbeat has to be sent. Increasing it will reduce the number of heart beats sent and when the Spark driver checks for the heartbeat every 2 minutes, there is more chance for failure.

To mitigate the issue "spark.network.timeout" can be increased. May to 300 s. Setting a very high value for spark.network.timeout is not recommended as that would delay the true failures.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.