https://community.databricks.com/t5/get-started-discussions/long-running-jobs-get-lost/m-p/66727#M7118 <P>Have you looked at the sql plan to see what the&nbsp; spark job 72 was doing?</P> Fri, 19 Apr 2024 11:16:46 GMT Lakshay 2024-04-19T11:16:46Z https://community.databricks.com/t5/get-started-discussions/long-running-jobs-get-lost/m-p/65993#M7116 <P>Hello,<BR />I tried to schedule a long running job and surprisingly it does seem to neither terminate (and thus does not let the cluster shut down), nor continue running, even though the state is still "Running":</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jenshumrich_0-1712742957610.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/6999i195F1D470C8FF458/image-size/medium/is-moderation-mode/true?v=v2&amp;px=400" role="button" title="jenshumrich_0-1712742957610.png" alt="jenshumrich_0-1712742957610.png" /></span><BR />But the truth is that the job has miserably failed:</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jenshumrich_2-1712743008070.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/7001i323F82383B72DB91/image-size/medium/is-moderation-mode/true?v=v2&amp;px=400" role="button" title="jenshumrich_2-1712743008070.png" alt="jenshumrich_2-1712743008070.png" /></span></P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="jenshumrich_3-1712743098546.png" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/7002i2EC39571384713A0/image-size/medium/is-moderation-mode/true?v=v2&amp;px=400" role="button" title="jenshumrich_3-1712743098546.png" alt="jenshumrich_3-1712743098546.png" /></span></P><P>Sadly thus the automatization is not working. Any hint would be appreciated</P><P>&nbsp;</P> Wed, 10 Apr 2024 10:00:23 GMT https://community.databricks.com/t5/get-started-discussions/long-running-jobs-get-lost/m-p/65993#M7116 jenshumrich 2024-04-10T10:00:23Z https://community.databricks.com/t5/get-started-discussions/long-running-jobs-get-lost/m-p/66645#M7117 <P><a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/102237">@jenshumrich</a>&nbsp;-&nbsp; There is not much information to decipher. However, can you please check if you have enough parallelism built for the task to execute. (spark.sql.shuffle.partitions and the no.of cores on the cluster) to begin with</P> Thu, 18 Apr 2024 22:39:01 GMT https://community.databricks.com/t5/get-started-discussions/long-running-jobs-get-lost/m-p/66645#M7117 shan_chandra 2024-04-18T22:39:01Z https://community.databricks.com/t5/get-started-discussions/long-running-jobs-get-lost/m-p/66727#M7118 <P>Have you looked at the sql plan to see what the&nbsp; spark job 72 was doing?</P> Fri, 19 Apr 2024 11:16:46 GMT https://community.databricks.com/t5/get-started-discussions/long-running-jobs-get-lost/m-p/66727#M7118 Lakshay 2024-04-19T11:16:46Z