I want to run an ETL job and when the job ends I would like to stop SparkSession to free my cluster's resources, by doing this I could avoid restarting the cluster, but when calling
spark.stop()
the job returns with status failed even though it has finished successfully, since the notebook automatically detach and the following error occurs:
The spark context has stopped and the driver is restarting. Your notebook will be automatically reattached.
I have read in this community's post that stopping the spark session is not recommended, but I do not understand the reason behind this. As far as I know, it is recommended to end the spark session after finishing the job in order the JVMs to close and free the resources.
Is there a recommended way to stop/restart SparkSession in a databricks job?
