org.apache.spark.SparkException: Job aborted due to stage failure during Model Training

[VeereshKH]

org.apache.spark.SparkException: Job aborted due to stage failure: Could not recover from a failed barrier ResultStage. Most recent failure reason: Stage failed because barrier task ResultTask(160, 13) finished unsuccessfully.

[Yeshwanth]

Hi @VeereshKH , I hope you are doing well.

Could you please share the complete error message and also confirm if you are using any spark configuration on the cluster?

We have seen "spark.databricks.pyspark.enableProcessIsolation" spark configuration was causing the problem in multiple scenarios. If you are using the same property, please try to remove it and rerun the code.

Please keep us posted with the results.

Thank you and wishing you an amazing day ahead!

[Lakshay]

Could you share the stage details where the issue happened?

[VeereshKH]

Stage failed because barrier task ResultTask(66, 1) finished unsuccessfully.

ExecutorLostFailure (executor 13 exited unrelated to the running tasks)

Reason: Executor decommission.org.apache.spark.rdd.RDD.collect(RDD.scala:1034) org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:260) org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380) py4j.Gateway.invoke(Gateway.java:295) py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) py4j.commands.CallCommand.execute(CallCommand.java:79) py4j.GatewayConnection.run(GatewayConnection.java:251) java.lang.Thread.run(Thread.java:748)

[Kaniz]

Hi @VeereshKH, The ExecutorLostFailure error message means one of the executors in the Apache Spark cluster has bee...2. This is a generic error message which can have more than one root cause12. Here are some possible reasons:

1. Out of Memory: The executor might be running out of memory1. You could try increasing the executor memory, but this is often a naive fix, and usually, the underl...3.
2. Executor is Busy: The executor might be too busy2.
3. Node Unavailability: This error could also indicate that a Spark task failed because a node terminat...4.

You might want to check the specific logs for more details. If the issue is due to memory, consider optimizing your Spark job to use less memory or increasing the memory allocated to your Spark executors. If the executor is busy, you might need to adjust the workload or increase the number of executors. If a node became unavailable, you might need to check your cluster’s health or configuration1243.