11-21-2019 08:40 PM
We are using the databricks 3 node cluster with 32 GB memory. It is working fine but some times it automatically throwing the error: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues.
11-21-2019 10:56 PM
Hi @naveen reddy
If you have 3 nodes with 32 GB memory specified each you have just 30 GB for everything else, the different overheads add up quick and it's entirely possible that this is too little and the executors get killed for hogging the memory.
Try using something like 24 GB per node or just play around with the values.
11-22-2019 10:13 AM
I have already tried with increasing and decreasing the memory, still no luck.
12-10-2019 11:55 AM
If your job fails follow this:
According to https://docs.databricks.com/jobs.html#jar-job-tips:"Job output, such as log output emitted to stdout, is subject to a 20MB size limit. If the total output has a larger size, the run will be canceled and marked as failed."
That was my problem, to "fix it" I've just set the logging level to ERRORval sc = SparkContext.getOrCreate(conf)sc.setLogLevel("ERROR")This workaround works for meI still get this ERROR messages but the job runs successfullyI hope it helps
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.