Re: Spark Driver Out of Memory Issue

chandan_a_v · ‎05-05-2022

Hi,

I am executing a simple job in Databricks for which I am getting below error. I increased the Driver size still I faced same issue.

Spark config :

from pyspark.sql import SparkSession

spark_session = SparkSession.builder.appName("Demand Forecasting").config("spark.yarn.executor.memoryOverhead", 2048).getOrCreate()

Driver and worker node type -r5.2xlarge

10 worker nodes.

Error Log:

Caused by: org.apache.spark.sql.execution.OutOfMemorySparkException: Size of broadcasted table far exceeds estimates and exceeds limit of spark.driver.maxResultSize=4294967296.

-werners- · ‎05-05-2022

looking at the error message you try to broadcast a large table. Remove the broadcast statement on the large table and you will be fine.

chandan_a_v · ‎05-08-2022

HI @Werner Stinckens ,

I am getting the above issue while writing a Spark DF as a parquet file to AWS S3. Not doing any broadcast join actually.

Thanks,

Chandan

Hubert-Dudek · ‎05-06-2022

In my opinion on databricks, you don't need to specify (spark_session = SparkSession.builder.appName("Demand Forecasting").config("spark.yarn.executor.memoryOverhead", 2048).getOrCreate()) and rest is as @Werner Stinckens said

My blog: https://databrickster.medium.com/

chandan_a_v · ‎05-08-2022

I am getting the above issue while writing a Spark DF as a parquet file to AWS S3. Not doing any broadcast join actually.

-werners- · ‎05-11-2022

As Hubert mentioned: you should not create a spark session on databricks, it is provided.

The fact you do not broadcast manually makes me think Spark uses a broadcastjoin.

There is a KB about issues with that:

https://kb.databricks.com/sql/bchashjoin-exceeds-bcjointhreshold-oom.html

Can you check if it is applicable?

chandan_a_v · ‎06-02-2022

Hi @Kaniz Fatma ,

Switching the runtime version to 10.4 fixed the issue for me.

Thanks,

Chandan

View solution in original post