ilir_nuredini
Honored Contributor

Hello @CzarR ,

From first glance it looks like offheap memory issue and thats why you would see a "GC overhead limit exceeded" error. 
Can you try enabling and adjusting the offheap memory size in the linked service where you define the cluster spark configurations and apply these configs:

"spark_conf": {
  "spark.memory.offHeap.enabled": "true",
  "spark.memory.offHeap.size": "36g"
}

Hope that helps. Let me know how it went and we can look into possible different options.

Best, Ilir

View solution in original post