https://community.databricks.com/t5/data-engineering/how-to-stop-sparksession-within-notebook-without-errr/m-p/36157#M26060 <P><SPAN>I want to run an ETL job and when the job ends&nbsp;</SPAN>I would like to stop SparkSession to free my cluster's resources, by doing this I could avoid restarting the cluster, but when calling</P><P>&nbsp;</P><LI-CODE lang="markup">spark.stop()</LI-CODE><P>&nbsp;</P><P>the job returns with status <STRONG>failed </STRONG>even though it has finished successfully, since the notebook automatically detach and the following error occurs:</P><P><EM>The spark context has stopped and the driver is restarting. Your notebook will be automatically reattached.&nbsp;</EM></P><P>I have read in <A href="https://community.databricks.com/t5/data-engineering/few-things-you-should-not-do-in-databricks/m-p/20866#M14127" target="_self">this community's post</A> that stopping the spark session is not recommended, but I do not understand the reason behind this. As far as I know, it is recommended to end the spark session after finishing the job in order the JVMs to close and free the resources.&nbsp;</P><P>Is there a recommended way to stop/restart SparkSession in a databricks job?</P> Thu, 29 Jun 2023 10:02:28 GMT gkrilis 2023-06-29T10:02:28Z https://community.databricks.com/t5/data-engineering/how-to-stop-sparksession-within-notebook-without-errr/m-p/36157#M26060 <P><SPAN>I want to run an ETL job and when the job ends&nbsp;</SPAN>I would like to stop SparkSession to free my cluster's resources, by doing this I could avoid restarting the cluster, but when calling</P><P>&nbsp;</P><LI-CODE lang="markup">spark.stop()</LI-CODE><P>&nbsp;</P><P>the job returns with status <STRONG>failed </STRONG>even though it has finished successfully, since the notebook automatically detach and the following error occurs:</P><P><EM>The spark context has stopped and the driver is restarting. Your notebook will be automatically reattached.&nbsp;</EM></P><P>I have read in <A href="https://community.databricks.com/t5/data-engineering/few-things-you-should-not-do-in-databricks/m-p/20866#M14127" target="_self">this community's post</A> that stopping the spark session is not recommended, but I do not understand the reason behind this. As far as I know, it is recommended to end the spark session after finishing the job in order the JVMs to close and free the resources.&nbsp;</P><P>Is there a recommended way to stop/restart SparkSession in a databricks job?</P> Thu, 29 Jun 2023 10:02:28 GMT https://community.databricks.com/t5/data-engineering/how-to-stop-sparksession-within-notebook-without-errr/m-p/36157#M26060 gkrilis 2023-06-29T10:02:28Z https://community.databricks.com/t5/data-engineering/how-to-stop-sparksession-within-notebook-without-errr/m-p/47777#M28206 <P>Please refer to this&nbsp;<A href="https://kb.databricks.com/en_US/jobs/job-fails-spark-finishes?_gl=1*1gh9veh*_gcl_au*MTYwNjYxNTA1MS4xNjk2MjE0Mzk5*_ga*MTIxMzMyNzkyMy4xNjk2MjE0NDA4*_ga_PQSEQ3RZQC*MTY5NjI5OTIxNC4yLjEuMTY5NjI5OTQ3NS42MC4wLjA.*rs_ga*ODM2MjQyODgtZDI5Mi00ZTJhLTliMjgtMTc0OGQyYjZlYWQw*rs_ga_PQSEQ3RZQC*MTY5NjI5OTIxNS4zLjEuMTY5NjI5OTQ3Ny4yMi4wLjA." target="_blank">Job fails, but Apache Spark tasks finish - Databricks</A></P> Tue, 03 Oct 2023 02:25:37 GMT https://community.databricks.com/t5/data-engineering/how-to-stop-sparksession-within-notebook-without-errr/m-p/47777#M28206 PremadasV 2023-10-03T02:25:37Z