https://community.databricks.com/t5/data-engineering/org-apache-spark-sparkexception-job-aborted-due-to-stage-failure/m-p/123274#M46972 <P>Hi everyone,<BR />I'm using a serverless cluster and encountering an issue where my code runs fine when executed cell-by-cell in a notebook, but fails with a memory error when executed as a job. Interestingly, the same job runs successfully in our dev environment using the same dataset.</P><P>Has anyone experienced a similar issue or have suggestions on how to resolve this? Any help would be appreciated.</P> Mon, 30 Jun 2025 11:24:41 GMT Parth2692 2025-06-30T11:24:41Z https://community.databricks.com/t5/data-engineering/org-apache-spark-sparkexception-job-aborted-due-to-stage-failure/m-p/123274#M46972 <P>Hi everyone,<BR />I'm using a serverless cluster and encountering an issue where my code runs fine when executed cell-by-cell in a notebook, but fails with a memory error when executed as a job. Interestingly, the same job runs successfully in our dev environment using the same dataset.</P><P>Has anyone experienced a similar issue or have suggestions on how to resolve this? Any help would be appreciated.</P> Mon, 30 Jun 2025 11:24:41 GMT https://community.databricks.com/t5/data-engineering/org-apache-spark-sparkexception-job-aborted-due-to-stage-failure/m-p/123274#M46972 Parth2692 2025-06-30T11:24:41Z https://community.databricks.com/t5/data-engineering/org-apache-spark-sparkexception-job-aborted-due-to-stage-failure/m-p/123695#M47055 <P>Hello&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/168824">@Parth2692</a>!</P> <P>It’s possible that your dev and prod environments have different serverless configurations, which could explain the difference in behavior.</P> <P>You can try <A href="https://learn.microsoft.com/en-us/azure/databricks/compute/serverless/dependencies#high-memory" target="_blank">increasing the notebook memory</A> by switching from Standard to High in the Environment side panel. However, note&nbsp;that this doesn’t affect the Spark executor memory, which can’t be manually configured when using serverless compute.</P> <P>If the issue persists, optimize your Spark job to reduce memory usage by splitting large jobs into smaller tasks, avoiding unnecessary caching, or adjusting how the data is partitioned.</P> Wed, 02 Jul 2025 14:01:11 GMT https://community.databricks.com/t5/data-engineering/org-apache-spark-sparkexception-job-aborted-due-to-stage-failure/m-p/123695#M47055 Advika 2025-07-02T14:01:11Z