Hi

I have around 20 million records in my DF, and want to save it in HORIZINTAL SQL DB.

This is error:

org.apache.spark.SparkException: Job aborted due to stage failure: A shuffle map stage with indeterminate output was failed and retried. However, Spark cannot rollback the ResultStage 1525 to re-process the input data, and has to fail this job. Please eliminate the indeterminacy by checkpointing the RDD before repartition and try again.

Here is my code:

df.write.format("jdbc").options( **DB_PROPS, **extra_options, dbtable=table, truncate=truncate).mode(mode).save()

Any opinion what can go wrong?

Regards