topic Re: Spark Driver Out of Memory Issue in Data Engineering

Spark Driver Out of Memory Issue

chandan_a_v — Fri, 06 May 2022 06:23:48 GMT

Hi,

I am executing a simple job in Databricks for which I am getting below error. I increased the Driver size still I faced same issue.

Spark config :

from pyspark.sql import SparkSession

spark_session = SparkSession.builder.appName("Demand Forecasting").config("spark.yarn.executor.memoryOverhead", 2048).getOrCreate()

Driver and worker node type -r5.2xlarge

10 worker nodes.

Error Log:

Caused by: org.apache.spark.sql.execution.OutOfMemorySparkException: Size of broadcasted table far exceeds estimates and exceeds limit of spark.driver.maxResultSize=4294967296.

Re: Spark Driver Out of Memory Issue

-werners- — Fri, 06 May 2022 06:54:49 GMT

looking at the error message you try to broadcast a large table. Remove the broadcast statement on the large table and you will be fine.

Re: Spark Driver Out of Memory Issue

Hubert-Dudek — Fri, 06 May 2022 16:04:50 GMT

In my opinion on databricks, you don't need to specify (spark_session = SparkSession.builder.appName("Demand Forecasting").config("spark.yarn.executor.memoryOverhead", 2048).getOrCreate()) and rest is as @Werner Stinckens said

Re: Spark Driver Out of Memory Issue

chandan_a_v — Sun, 08 May 2022 19:05:33 GMT

HI @Werner Stinckens ,

I am getting the above issue while writing a Spark DF as a parquet file to AWS S3. Not doing any broadcast join actually.

Thanks,

Chandan

Re: Spark Driver Out of Memory Issue

chandan_a_v — Sun, 08 May 2022 19:05:48 GMT

I am getting the above issue while writing a Spark DF as a parquet file to AWS S3. Not doing any broadcast join actually.

Re: Spark Driver Out of Memory Issue

-werners- — Wed, 11 May 2022 13:25:58 GMT

As Hubert mentioned: you should not create a spark session on databricks, it is provided.

The fact you do not broadcast manually makes me think Spark uses a broadcastjoin.

There is a KB about issues with that:

https://kb.databricks.com/sql/bchashjoin-exceeds-bcjointhreshold-oom.html

Can you check if it is applicable?

Re: Spark Driver Out of Memory Issue

chandan_a_v — Thu, 02 Jun 2022 08:51:38 GMT

Hi @Kaniz Fatma ,

Switching the runtime version to 10.4 fixed the issue for me.

Thanks,

Chandan