Can you access the executor logs? When you cluster is up and running, you can access the executor's logs. For example, the error shows:

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3) (10.139.64.6 executor 0): org.apache.spark.SparkException: Exception thrown in awaitResult:

Go to the Executor 0 and check why it failed