topic Re: Concurrent Jobs - The spark driver has stopped unexpectedly! in Data Engineering

Concurrent Jobs - The spark driver has stopped unexpectedly!

uzairm — Mon, 27 Feb 2023 20:29:48 GMT

Hi, I am running concurrent notebooks in concurrent workflow jobs in job compute cluster c5a.8xlarge with 5-7 worker nodes. Each job has 100 concurrent child notebooks and there are 10 job instances. 8/10 jobs gives the error the spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached.

How can I resolve that?

Re: Concurrent Jobs - The spark driver has stopped unexpectedly!

daniel_sahal — Tue, 28 Feb 2023 06:22:43 GMT

@uzair mustafa

Check the Ganglia for performance related issues (maybe it's getting OOM?).

Re: Concurrent Jobs - The spark driver has stopped unexpectedly!

uzairm — Tue, 28 Feb 2023 06:49:51 GMT

Hi,

I have been checking Ganglia. Free Space is about 300GB available.

Re: Concurrent Jobs - The spark driver has stopped unexpectedly!

daniel_sahal — Wed, 01 Mar 2023 06:41:41 GMT

@uzair mustafa

It's hard to answer without digging into the logs and code.

Re: Concurrent Jobs - The spark driver has stopped unexpectedly!

uzairm — Wed, 01 Mar 2023 07:12:00 GMT

I have identified the issue. The driver memory is exhausting and the worker nodes are not coming into play...

Re: Concurrent Jobs - The spark driver has stopped unexpectedly!

daniel_sahal — Wed, 01 Mar 2023 07:34:28 GMT

@uzair mustafa

So basically, that's what I expected, OOM 🙂

It's good that you were able to find an issue.

Re: Concurrent Jobs - The spark driver has stopped unexpectedly!

Anonymous — Mon, 13 Mar 2023 04:53:42 GMT

Hi @uzair mustafa

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you.

Cheers!

Re: Concurrent Jobs - The spark driver has stopped unexpectedly!

uzairm — Mon, 13 Mar 2023 05:09:20 GMT

Hi Vidula,
I still face the issue and it has not been resolved. It would be great if some one helps me.

Re: Concurrent Jobs - The spark driver has stopped unexpectedly!

JKR — Wed, 26 Apr 2023 08:10:58 GMT

Did you solve this issue ? I'm in similar situation.

Re: Concurrent Jobs - The spark driver has stopped unexpectedly!

uzairm — Wed, 26 Apr 2023 10:24:19 GMT

@Jeetash Kumar I identified the issue, which was the driver memory was getting exhausted. I fine tuned my code so that lesser operations are done on the driver side and I reduced the concurrency of my tasks. This answer is based on my use case.

Re: Concurrent Jobs - The spark driver has stopped unexpectedly!

uzairm — Wed, 26 Apr 2023 10:25:02 GMT

You can take a look at your driver memory by looking at the Ganglia UI, monitor it as it your cluster runs..

Re: Concurrent Jobs - The spark driver has stopped unexpectedly!

JKR — Wed, 26 Apr 2023 11:09:13 GMT

Yes, I'm monitoring driver memory in Ganglia (Attaching SS of the driver node).

what might be the list of operations are done on the driver side which I need to avoid ?

Re: Concurrent Jobs - The spark driver has stopped unexpectedly!

uzairm — Wed, 26 Apr 2023 11:11:27 GMT

Operations like collect() select() are done on driver node. All aggregations. Avoid those.