Long running jobs get lost

[jenshumrich]

Hello,
I tried to schedule a long running job and surprisingly it does seem to neither terminate (and thus does not let the cluster shut down), nor continue running, even though the state is still "Running":

But the truth is that the job has miserably failed:

Sadly thus the automatization is not working. Any hint would be appreciated

[shan_chandra]

@jenshumrich -  There is not much information to decipher. However, can you please check if you have enough parallelism built for the task to execute. (spark.sql.shuffle.partitions and the no.of cores on the cluster) to begin with

[Lakshay]

Have you looked at the sql plan to see what the  spark job 72 was doing?