Spark out of memory error.You can resolve this error by increasing the size of cluster in Databricks.

[SS2]

Spark out of memory error.

You can resolve this error by increasing the size of cluster in Databricks.

[karthik_p]

@S S​ every time cluster increase may not be good solution, based on scenarios it gets changed

1. sometimes we may need to tweak code
2. sometimes we may need to add memory parameters
3. Based on ganglia metrics we can get more information

[NhatHoang]

Hi guys,

I agree​, it is better if you improve you code rather than increase the size of cluster. You can config the number of partitions.

[Shalabh007]

Directly jumping on solution to inc the cluster size is not advisible. I found this nicely written blog what could be potential reason and some initial steps to resolve the OOM error in spark.

https://medium.com/swlh/spark-oom-error-closeup-462c7a01709d

[DK03]

Adding some more points to @karthik p​ 's answer.

1. Use kryo serializer instead of java serializer.
2. Use an optimised garbage collector such as G1GC.
3. Use partitioning wisely on a field.