topic Re: Error while optimizing the table . Failure of InSet.sql for UTF8String collection in Data Engineering

Error while optimizing the table . Failure of InSet.sql for UTF8String collection

nagini_sitarama — Fri, 02 Dec 2022 21:56:23 GMT

count of the table : 1125089 for october month data , So I am optimizing the table.

optimize table where batchday >="2022-10-01" and batchday<="2022-10-31"

I am getting error like : GC overhead limit exceeded

at org.apache.spark.unsafe.types.UTF8String.fromBytes(UTF8String.java:136)

I have increased my driver and executor worker nodes from 2 worker nodes to 10 worker nodes with memory size increased from 32 GB .

When i run other optimize on other batchday i am not finding any issue.

Could you tell me why UTFString.fromBytes exception comes during optimizing a partition of data?

UmaMahesh1 — Sat, 03 Dec 2022 15:23:36 GMT

Hi @Nagini Sitaraman

Can you try forcing the GC to unload the objects in the driver's memory. That should solve your issue.

code to do that :

spark.catalog.clearCache()

for (id, rdd) in spark.sparkContext._jsc.getPersistentRDDs().items():

rdd.unpersist()

print("Unloaded {} rdd".format(id))

Cheers

nagini_sitarama — Thu, 08 Dec 2022 17:11:41 GMT

Hi Uma,

I am running just a optimize sql script .

optimize table where batchday >="2022-10-01" and batchday<="2022-10-31"

In this case how to handle this .

There are 93 files present in this batch .

It takes more than 5 hours to optimize .

Priyanka_Biswas — Mon, 30 Jan 2023 16:41:02 GMT

Hi @Nagini Sitaraman To understand the issue better I would like to get some more information.

Does the error occur at the driver side or executor side?

Can you please share the full error stack trace?

You may need to check the spark UI to find where the bottleneck is. e.g. which phase causes the issue, it’s memory or other issue?