org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 69 tasks (4.0 GB) is bigger than spark.driver.maxResultSize (4.0 GB)

sachinmkp1
New Contributor II

‎08-23-2021 07:48 AM

set spark.conf.set("spark.driver.maxResultSize", "20g")

get spark.conf.get("spark.driver.maxResultSize") // 20g which is expected in notebook , I did not do in cluster level setting

still getting 4g while executing the spark job , why?

because of this job is getting failed.

Labels:
0 Kudos