I created some ETL using DataFrames in python. It used to run 180 sec. But it is not taking ~ 1200 sec. I have been changing it, so it could be something that I introduced, or something in the environment.

Part of the process is appending results into a file on S3.

I a looking at Apache Jobs and I cannot see that any of them is active.

While I was writing this, I got: org.apache.spark.SparkException: Job aborted.

Command took 1274.63s -- by xxxxxxxx@gmail.com

at 8/4/2016, 12:44:17 PM on def4 (150 GB)

I have attached output that I got:

command-output.txt

I assume that I should be able to see in Spark UI what is active. I was surprised that Active Tasks on all executors was 0. Should I look at something else?

I tried to restart the cluster, but it was the same before and after. I used the same version of Spark 1.6.2 (Hadoop 2).