Hi Raghuram,

I checked the shard and noticed a few things. The executors have died and restarted on the cluster, and one of them continues to die likely due to out of memory errors. I'd have to dig deeper into why this occurred and it varies depending on the workload being run on the cluster. If you click on the Spark UI -> Executors tab, you can see the executor ID has crashed multiple times as the executor IDs increase by increments of 1.

In the UI, I also see that multiple SQLContexts are being created which isn't necessary. The sqlContext and sc (SparkContext) are already created on the clusters. If you would like to assign a new variable to them use the following code example:

val sqlContext = SQLContext.getOrCreate(sc)

You can access the sqlContext by using "sqlContext" variable, and the SparkContext using the "sc" variable across the notebooks.

I'd recommend restarting this cluster to get things back in a good state.

Let me know if the issue persists after a restart.