topic Re: org.apache.spark.SparkException: Job aborted due to stage failure: in Get Started Discussions

org.apache.spark.SparkException: Job aborted due to stage failure:

Manmohan_Nayak — Sun, 04 Feb 2024 17:59:58 GMT

I have around 20 million records in my DF, and want to save it in HORIZINTAL SQL DB.

This is error:

org.apache.spark.SparkException: Job aborted due to stage failure: A shuffle map stage with indeterminate output was failed and retried. However, Spark cannot rollback the ResultStage 1525 to re-process the input data, and has to fail this job. Please eliminate the indeterminacy by checkpointing the RDD before repartition and try again.

Here is my code:

df.write.format("jdbc").options( **DB_PROPS, **extra_options, dbtable=table, truncate=truncate).mode(mode).save()

Any opinion what can go wrong?

Regards

Re: org.apache.spark.SparkException: Job aborted due to stage failure:

aniketg — Wed, 07 Feb 2024 10:53:39 GMT

@Manmohan_Nayak If the resolution worked for you?
I am facing the same error from last couple of days for the job which was working earlier

Re: org.apache.spark.SparkException: Job aborted due to stage failure:

Dusan — Fri, 24 May 2024 08:56:46 GMT

facing same issue since we moved from Spark 3.2.1 (databricks 10.4) to Spark 3.3.2 (databricks 12.2), how come we have seen this problem before, now we do.. is it Spark related or Databricks related (autoscaling?)

Re: org.apache.spark.SparkException: Job aborted due to stage failure:

VZLA — Tue, 04 Jun 2024 12:49:58 GMT

If there are any failures which may lead to a stage retry, but retrying the stage translates into potentially having an inconsistent result (indeterminacy) then this exception is raised. The exception is raised in newer version where the validation is performed, likely unavailable in DBR 10.4 and older versions.

To address the problem, you may as per the error message, checkpoint the DF before the indeterminacy is introduce.

This can be commonly seen in scenarios where there are nodes lost, for example due to spot instance termination, or similar events, not fully sure about a scaling down event, but could also be another reason.