topic Re: Spark out of memory error.You can resolve this error by increasing the size of cluster in Databricks. in Data Engineering

Spark out of memory error.You can resolve this error by increasing the size of cluster in Databricks.

SS2 — Tue, 29 Nov 2022 20:08:49 GMT

Spark out of memory error.

You can resolve this error by increasing the size of cluster in Databricks.

Re: Spark out of memory error.You can resolve this error by increasing the size of cluster in Databricks.

karthik_p — Tue, 29 Nov 2022 21:11:55 GMT

@S S every time cluster increase may not be good solution, based on scenarios it gets changed

sometimes we may need to tweak code
sometimes we may need to add memory parameters
Based on ganglia metrics we can get more information

Re: Spark out of memory error.You can resolve this error by increasing the size of cluster in Databricks.

NhatHoang — Wed, 30 Nov 2022 04:00:07 GMT

Hi guys,

I agree, it is better if you improve you code rather than increase the size of cluster. You can config the number of partitions.

Re: Spark out of memory error.You can resolve this error by increasing the size of cluster in Databricks.

Shalabh007 — Wed, 30 Nov 2022 06:41:39 GMT

Directly jumping on solution to inc the cluster size is not advisible. I found this nicely written blog what could be potential reason and some initial steps to resolve the OOM error in spark.

https://medium.com/swlh/spark-oom-error-closeup-462c7a01709d

Re: Spark out of memory error.You can resolve this error by increasing the size of cluster in Databricks.

DK03 — Wed, 30 Nov 2022 10:00:27 GMT

Adding some more points to @karthik p 's answer.

Use kryo serializer instead of java serializer.
Use an optimised garbage collector such as G1GC.
Use partitioning wisely on a field.