topic Re: Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages. Driver stacktrace in Data Engineering

Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages. Driver stacktrace

naveenreddy1 — Fri, 22 Nov 2019 04:40:58 GMT

We are using the databricks 3 node cluster with 32 GB memory. It is working fine but some times it automatically throwing the error: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues.

Re: Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages. Driver stacktrace

shyam_9 — Fri, 22 Nov 2019 06:56:30 GMT

Hi @naveen reddy

If you have 3 nodes with 32 GB memory specified each you have just 30 GB for everything else, the different overheads add up quick and it's entirely possible that this is too little and the executors get killed for hogging the memory.

Try using something like 24 GB per node or just play around with the values.

Re: Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages. Driver stacktrace

naveenreddy1 — Fri, 22 Nov 2019 18:13:35 GMT

I have already tried with increasing and decreasing the memory, still no luck.

Re: Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages. Driver stacktrace

RodrigoDe_Freit — Tue, 10 Dec 2019 19:55:58 GMT

If your job fails follow this:

According to https://docs.databricks.com/jobs.html#jar-job-tips:

"Job output, such as log output emitted to stdout, is subject to a 20MB size limit. If the total output has a larger size, the run will be canceled and marked as failed."

That was my problem, to "fix it" I've just set the logging level to ERROR

val sc = SparkContext.getOrCreate(conf)

sc.setLogLevel("ERROR")

This workaround works for me

I still get this ERROR messages but the job runs successfully

I hope it helps

Re: Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or netwo

elguitar — Fri, 08 Nov 2024 06:12:45 GMT

The documentation has changed, and that 20MB log limit is now documented in https://docs.databricks.com/en/jobs/how-to/use-jars-in-workflows.html#output-size-limits-for-jar-jobs.