Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages. Driver stacktrace

[naveenreddy1]

We are using the databricks 3 node cluster with 32 GB memory. It is working fine but some times it automatically throwing the error: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues.

[shyam_9]

Hi @naveen reddy

If you have 3 nodes with 32 GB memory specified each you have just 30 GB for everything else, the different overheads add up quick and it's entirely possible that this is too little and the executors get killed for hogging the memory.

Try using something like 24 GB per node or just play around with the values.

[naveenreddy1]

I have already tried with increasing and decreasing the memory, still no luck.

[RodrigoDe_Freit]

If your job fails follow this:

According to https://docs.databricks.com/jobs.html#jar-job-tips:

"Job output, such as log output emitted to stdout, is subject to a 20MB size limit. If the total output has a larger size, the run will be canceled and marked as failed."

That was my problem, to "fix it" I've just set the logging level to ERRORval sc = SparkContext.getOrCreate(conf)sc.setLogLevel("ERROR")This workaround works for meI still get this ERROR messages but the job runs successfully

I hope it helps

[elguitar]

The documentation has changed, and that 20MB log limit is now documented in https://docs.databricks.com/en/jobs/how-to/use-jars-in-workflows.html#output-size-limits-for-jar-job....