Requested array size exceeds VM limit when saving to feature table

pcriado — Wed, 07 Jun 2023 21:09:05 GMT

Hi, I'm trying to process a small dataset (less than 300 Mb) composed by five queries that run with spark. The end result of those queries is parsed using python and merged into a data frame. Then I try to write this to a delta lake table using features:

parsedData.write.format('delta').mode('overwrite').option("mergeSchema", "true").save('/mnt/features/dev_customer_account_info')

This single line of code above always causes a memory spike leading to a crash (60 Gb), regardless of the size of parsedData.

The cluster is configured as:

1 Driver 61 GB Memory, 8 Cores

Runtime 11.3.x-cpu-ml-scala2.12

The error looks like this:

java.lang.OutOfMemoryError: Requested array size exceeds VM limit
	at java.lang.StringCoding.encode(StringCoding.java:350)
	at java.lang.String.getBytes(String.java:941)
	at org.apache.spark.unsafe.types.UTF8String.fromString(UTF8String.java:163)
	at org.apache.spark.sql.catalyst.expressions.StructsToJson.getAndReset$1(jsonExpressions.scala:893)
	at org.apache.spark.sql.catalyst.expressions.StructsToJson.$anonfun$converter$5(jsonExpressions.scala:904)
	at org.apache.spark.sql.catalyst.expressions.StructsToJson$$Lambda$12421/1187286213.apply(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.StructsToJson.nullSafeEval(jsonExpressions.scala:947)
	at org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:671)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificMutableProjection.apply(Unknown Source)
	at org.apache.spark.sql.execution.python.EvalPythonExec.$anonfun$doExecute$10(EvalPythonExec.scala:127)
	at org.apache.spark.sql.execution.python.EvalPythonExec$$Lambda$12407/1574333163.apply(Unknown Source)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at org.apache.spark.util.GroupedAsArrayIterator.next(GroupedAsArrayIterator.scala:45)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at scala.collection.Iterator.foreach(Iterator.scala:943)
	at scala.collection.Iterator.foreach$(Iterator.scala:943)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
	at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:464)
	at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$1.writeIteratorToStream(PythonUDFRunner.scala:55)
	at org.apache.spark.api.python.BasePythonRunner$WriterThread.$anonfun$run$1(PythonRunner.scala:573)
	at org.apache.spark.api.python.BasePythonRunner$WriterThread$$Lambda$11996/626269711.apply(Unknown Source)
	at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:2340)
	at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:365)

I've tried to run the queries with paging, to reduce the amount of data that should be saved to the table, probably reduce it to as little as 100 Mb, but this part always consumes all available ram and crashes.

The contents of the data frame are pretty standard. I'm at a loss here about what could be done. I'd really appreciate any comments, thoughts or ideas.

Thank you very much

Re: Requested array size exceeds VM limit when saving to feature table

Anonymous — Fri, 16 Jun 2023 06:03:56 GMT

Hi @Pablo Criado

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question. Thanks.

Re: Requested array size exceeds VM limit when saving to feature table

pcriado — Fri, 16 Jun 2023 12:30:58 GMT

Hello, we have recently found that it's my user in particular that casues the memory issue. Two other users in my organization can run the same notebook without problems, but my user consistenly consumes all available ram and crashes the cluster... and I have absolutely no idea how something like that could happen.

topic Requested array size exceeds VM limit when saving to feature table in Data Engineering

Requested array size exceeds VM limit when saving to feature table

Re: Requested array size exceeds VM limit when saving to feature table

Re: Requested array size exceeds VM limit when saving to feature table