Concurrent Jobs - The spark driver has stopped unexpectedly!

[uzairm]

Hi, I am running concurrent notebooks in concurrent workflow jobs in job compute cluster c5a.8xlarge with 5-7 worker nodes. Each job has 100 concurrent child notebooks and there are 10 job instances. 8/10 jobs gives the error the spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached.

How can I resolve that?

[daniel_sahal]

@uzair mustafa​ 

Check the Ganglia for performance related issues (maybe it's getting OOM?).

[uzairm]

Hi,

I have been checking Ganglia. Free Space is about 300GB available.

[daniel_sahal]

@uzair mustafa​ 

It's hard to answer without digging into the logs and code.

[uzairm · Accepted solution]

I have identified the issue. The driver memory is exhausting and the worker nodes are not coming into play...

[daniel_sahal]

@uzair mustafa​ 

So basically, that's what I expected, OOM 🙂

It's good that you were able to find an issue.

[Anonymous]

Hi @uzair mustafa​ 

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you. 

Cheers!

[uzairm]

Hi Vidula,
I still face the issue and it has not been resolved. It would be great if some one helps me.

[JKR]

Did you solve this issue ? I'm in similar situation.

[uzairm]

@Jeetash Kumar​ I identified the issue, which was the driver memory was getting exhausted. I fine tuned my code so that lesser operations are done on the driver side and I reduced the concurrency of my tasks. This answer is based on my use case.

[uzairm]

You can take a look at your driver memory by looking at the Ganglia UI, monitor it as it your cluster runs..

[JKR]

Yes, I'm monitoring driver memory in Ganglia (Attaching SS of the driver node).

what might be the list of operations are done on the driver side which I need to avoid ?

[uzairm]

Operations like collect() select() are done on driver node. All aggregations. Avoid those.