org.apache.spark.SparkException: Job aborted due to stage failure: org.apache.spark.memory.SparkOutO

[Parth2692]

Hi everyone,
I'm using a serverless cluster and encountering an issue where my code runs fine when executed cell-by-cell in a notebook, but fails with a memory error when executed as a job. Interestingly, the same job runs successfully in our dev environment using the same dataset.

Has anyone experienced a similar issue or have suggestions on how to resolve this? Any help would be appreciated.

[Advika]

Hello @Parth2692!

It’s possible that your dev and prod environments have different serverless configurations, which could explain the difference in behavior.

You can try increasing the notebook memory by switching from Standard to High in the Environment side panel. However, note that this doesn’t affect the Spark executor memory, which can’t be manually configured when using serverless compute.

If the issue persists, optimize your Spark job to reduce memory usage by splitting large jobs into smaller tasks, avoiding unnecessary caching, or adjusting how the data is partitioned.