topic Re: Remote RPC client disassociated error in Data Engineering

Remote RPC client disassociated error

Alix — Mon, 21 Feb 2022 17:00:10 GMT

Hello,

I've been trying to submit a job to a transient cluster, but it is failing with this error :

Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 7) (10.139.64.5 executor 4): ExecutorLostFailure (executor 4 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.

the same job works fine on an interactive cluster with same specs (also the job is pretty tiny so I don't get that containers exceeding ...), I don't specify anything special at cluster creation, and I don't install any special libraries regarding spark ... I'm running out of idea on what could be the error, any clues ?

Thanks 🙂

Re: Remote RPC client disassociated error

Anonymous — Mon, 21 Feb 2022 19:11:35 GMT

Hello, @Alix Métivier - My name is Piper and I'm a moderator for Databricks. Welcome to the community and thank you for your question. We'll give it a while to see what your fellow members have to say. We'll circle back around if we need to.

Thanks in advance for your patience.

Re: Remote RPC client disassociated error

AmanSehgal — Tue, 22 Feb 2022 02:16:32 GMT

@Alix Métivier could you check the output of the entire notebook or of each cell?

There's a size limit on cell output and for entire notebook.

Re: Remote RPC client disassociated error

Alix — Tue, 22 Feb 2022 08:28:26 GMT

Hi @Aman Sehgal , I just had a run with the option spark.databricks.driver.disableScalaOutput set to true and the error is still there

I'm not using notebook but I'm using Runs submit with a java jar

Re: Remote RPC client disassociated error

AmanSehgal — Tue, 22 Feb 2022 12:06:28 GMT

What else can you grab from spark logs?

Re: Remote RPC client disassociated error

Alix — Tue, 22 Feb 2022 13:18:39 GMT

thats all the logs i get and theres not much help inside

Re: Remote RPC client disassociated error

Alix — Wed, 23 Feb 2022 08:42:36 GMT

issue was caused by the fact that I set spark.serializer", "org.apache.spark.serializer.KryoSerializer" and "spark.kryo.registrator" in the spark conf of the transient cluster

after removing them its working, does that mean that databricks does not support kryo with transient ?

Re: Remote RPC client disassociated error

shan_chandra — Wed, 11 May 2022 02:02:51 GMT

@Alix Métivier - The error is thrown from the user code (please investigate the jar file attached to the cluster).

at m80.dbruniv_0_1.dbruniv.tFixedFlowInput_1Process(dbruniv.java:941)

at m80.dbruniv_0_1.dbruniv.run(dbruniv.java:1654)

at m80.dbruniv_0_1.dbruniv.runJobInTOS(dbruniv.java

Re: Remote RPC client disassociated error

jose_gonzalez — Tue, 07 Jun 2022 16:16:49 GMT

Hi @Alix Métivier ,

Are you able to get the logs for executor 4? it seems like these logs are from the driver, not the executor.