cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Is it recommended to turn on Spark speculative execution permanently

brickster_2018
Esteemed Contributor

I had a job where the last step will get stuck forever. Turning on spark speculative execution did magic and resolved the issue.

Is it safe to turn on Spark speculative execution permanently.

1 ACCEPTED SOLUTION

Accepted Solutions

brickster_2018
Esteemed Contributor

It's not recommended to turn of Spark speculative execution permanently. For jobs where tasks are running slow or stuck because of transient network or storage issues, speculative execution can be very handy. However, it suppresses the actual problem and performs a retry of the task.

Speculative execution should be treated as a temporary workaround until finding the root cause of why the task or job is stuck.

Speculative execution can cause unnecessary task retries and can degrade the performance of jobs/stages where there is no true task stuck scenarios.

View solution in original post

1 REPLY 1

brickster_2018
Esteemed Contributor

It's not recommended to turn of Spark speculative execution permanently. For jobs where tasks are running slow or stuck because of transient network or storage issues, speculative execution can be very handy. However, it suppresses the actual problem and performs a retry of the task.

Speculative execution should be treated as a temporary workaround until finding the root cause of why the task or job is stuck.

Speculative execution can cause unnecessary task retries and can degrade the performance of jobs/stages where there is no true task stuck scenarios.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group