topic Re: java.lang.OutOfMemoryError: GC overhead limit exceeded. [ solved ] in Data Engineering

java.lang.OutOfMemoryError: GC overhead limit exceeded. [ solved ]

sarvesh — Tue, 23 Nov 2021 05:51:42 GMT

solution :-

i don't need to add any executor or driver memory all i had to do in my case was add this : - option("maxRowsInMemory", 1000).

Before i could n't even read a 9mb file now i just read a 50mb file without any error.

{

val df = spark.read

.format("com.crealytics.spark.excel").

option("maxRowsInMemory", 1000).

option("header", "true").

load("data/12file.xlsx")

}

I am trying to read a 8mb excel file,

i am getting this error.

i use intellij with spark 2.4.4

scala 2.12.12

and jdk 1.8

this is my code : -

val conf = new SparkConf()

.set("spark.driver.memory","4g")

.set("spark.executor.memory", "6g")

// .set("spark.executor.cores", "2")

val spark = SparkSession

.builder

.appName("trimTest")

.master("local[*]")

.config(conf)

.getOrCreate()

val df = spark.read

.format("com.crealytics.spark.excel").

option("header", "true").

load("data/12file.xlsx")

Now, these are my spark ui screenshots,

can you tell me what is the main issue and how can i increase the job executor memory.

stack :-

java.lang.OutOfMemoryError: GC overhead limit exceeded

at java.lang.Class.newReflectionData(Class.java:2511)

at java.lang.Class.reflectionData(Class.java:2503)

at java.lang.Class.privateGetDeclaredConstructors(Class.java:2660)

at java.lang.Class.getConstructor0(Class.java:3075)

at java.lang.Class.newInstance(Class.java:412)

at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:403)

at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)

at java.security.AccessController.doPrivileged(Native Method)

at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)

at sun.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:276)

at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:193)

at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:175)

at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:117)

at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:54)

at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)

at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83)

at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206)

at javax.management.StandardMBean.getAttribute(StandardMBean.java:372)

at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)

at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)

at com.sun.jmx.mbeanserver.MXBeanProxy$GetHandler.invoke(MXBeanProxy.java:122)

at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(MXBeanProxy.java:167)

at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:258)

at com.sun.proxy.$Proxy8.getMemoryUsed(Unknown Source)

at org.apache.spark.metrics.MBeanExecutorMetricType.getMetricValue(ExecutorMetricType.scala:67)

at org.apache.spark.metrics.SingleValueExecutorMetricType.getMetricValues(ExecutorMetricType.scala:46)

at org.apache.spark.metrics.SingleValueExecutorMetricType.getMetricValues$(ExecutorMetricType.scala:44)

at org.apache.spark.metrics.MBeanExecutorMetricType.getMetricValues(ExecutorMetricType.scala:60)

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded. [ solved ]

-werners- — Tue, 23 Nov 2021 07:05:56 GMT

I doubt it is the 8 MB file.

What happens if you do not set any memory parameter at all? (use the defaults)

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded. [ solved ]

sarvesh — Tue, 23 Nov 2021 07:40:04 GMT

it is an 8.5 mb xlsx file with 100k rows of data,

i get the same gc overhead limit exceeded error without addin any parameter

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded. [ solved ]

-werners- — Tue, 23 Nov 2021 07:56:21 GMT

My guess is indeed a config issue as in your spark script you don't seem to do any action (spark is lazy evaluated).

As you run Spark locally, chances are the JVM cannot allocate enough RAM for it to run succesfully.

Can you check the docs:

https://spark.apache.org/docs/2.4.4/tuning.html#garbage-collection-tuning

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded. [ solved ]

sarvesh — Tue, 23 Nov 2021 08:44:25 GMT

yes i just went through it, and from what i understood i need to increase the heap space, but increasing it at run time with intellij is not working.

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded. [ solved ]

-werners- — Tue, 23 Nov 2021 09:02:37 GMT

can you try with local[2] instead of local[*]?

And also beef up the driver memory to like 90% of your RAM.

As you run in local mode, the driver and the executor all run in the same process which is controlled by driver memory.

So you can skip the executor params.

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded. [ solved ]

sarvesh — Tue, 23 Nov 2021 09:38:14 GMT

i did it is still the same, there is something else that i am missing here, and my memory consumed was 7gb out of 8gb available right now.

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded. [ solved ]

-werners- — Tue, 23 Nov 2021 09:44:30 GMT

you don't do anything else but the spark.read.excel, right?

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded. [ solved ]

sarvesh — Tue, 23 Nov 2021 10:15:43 GMT

i am doing df.show(),

nothing else

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded. [ solved ]

-werners- — Tue, 23 Nov 2021 10:18:58 GMT

weird, when I run spark locally, I just install it, do not configure any executor and it just works.

Did you define any executors by any chance?

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded. [ solved ]

sarvesh — Tue, 23 Nov 2021 10:49:26 GMT

no i did n't i have also just installed it, i think this is my machines issue or something which i have not done right.

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded. [ solved ]

Hubert-Dudek — Tue, 23 Nov 2021 11:07:12 GMT

can you try without:

.set("spark.driver.memory","4g")

.set("spark.executor.memory", "6g")

It is clearly show that there is no 4gb free on driver and 6gb free on executor (you can share hardware cluster details also).

You can not also allocate 100% for spark usually as there is also other processes.

Automatic settings are recommended.

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded. [ solved ]

sarvesh — Tue, 23 Nov 2021 12:00:18 GMT

i tried to read it without these configs i got the same error ( gc overhead limit ), and i am running it locally these are my specifications.

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded. [ solved ]

Hubert-Dudek — Tue, 23 Nov 2021 12:50:54 GMT

It seems that you have only 8GB ram (probably 4-6 GB is needed for system at least) but you allocate 10GB for spark (4 GB driver + 6 GB executor).

You can allocate max in my opinion 2GB all together if your RAM is 8 GB. Maybe even 1GB as there can be also spikes in system processes.

Easier would be with docker as than you allocate your machine permanent number of ram and than spark can consume exact amount.

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded. [ solved ]

-werners- — Tue, 23 Nov 2021 13:28:10 GMT

Yea that will be it. 8GB can run spark, but I'd go for no more than 3GB, 2GB on the safe side.

It looks like an ubuntu install so that is not as resource hungry than windows but 8GB is not much.

For tinkering around, I always go for Docker (or a VM).

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded. [ solved ]

sarvesh — Thu, 25 Nov 2021 11:12:58 GMT

Thank You i just found a solution, and i have mentioned it in my question to, while reading my file all i had to do was add this,

option("maxRowsInMemory", 1000).

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded. [ solved ]

sarvesh — Thu, 25 Nov 2021 11:13:11 GMT

found a way

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded. [ solved ]

jose_gonzalez — Mon, 29 Nov 2021 19:07:30 GMT

Hi @sarvesh singh ,

Could you mark the "best" solution to your question please? it will help in case other community members have the se issue in the future.

Thank you

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded. [ solved ]

sarvesh — Wed, 01 Dec 2021 13:12:05 GMT

done