https://community.databricks.com/t5/data-engineering/is-it-recommended-to-turn-on-spark-speculative-execution/m-p/20472#M13811 <P>I had a job where the last step will get stuck forever. Turning on spark speculative execution did magic and resolved the issue. </P><P></P><P>Is it safe to turn on Spark speculative execution permanently. </P> Thu, 24 Jun 2021 17:38:28 GMT brickster_2018 2021-06-24T17:38:28Z https://community.databricks.com/t5/data-engineering/is-it-recommended-to-turn-on-spark-speculative-execution/m-p/20472#M13811 <P>I had a job where the last step will get stuck forever. Turning on spark speculative execution did magic and resolved the issue. </P><P></P><P>Is it safe to turn on Spark speculative execution permanently. </P> Thu, 24 Jun 2021 17:38:28 GMT https://community.databricks.com/t5/data-engineering/is-it-recommended-to-turn-on-spark-speculative-execution/m-p/20472#M13811 brickster_2018 2021-06-24T17:38:28Z https://community.databricks.com/t5/data-engineering/is-it-recommended-to-turn-on-spark-speculative-execution/m-p/20473#M13812 <P>It's not recommended to turn of Spark speculative execution permanently. For jobs where tasks are running slow or stuck because of transient network or storage issues, speculative execution can be very handy. However, it suppresses the actual problem and performs a retry of the task. </P><P>Speculative execution should be treated as a temporary workaround until finding the root cause of why the task or job is stuck. </P><P>Speculative execution can cause unnecessary task retries and can degrade the performance of jobs/stages where there is no true task stuck scenarios. </P><P></P> Thu, 24 Jun 2021 17:41:35 GMT https://community.databricks.com/t5/data-engineering/is-it-recommended-to-turn-on-spark-speculative-execution/m-p/20473#M13812 brickster_2018 2021-06-24T17:41:35Z