Databricks Community

GC-James · ‎02-23-2022

Sometimes when I am running R code in a Databricks notebook I am given this error. The cell I am running fails, and my whole R 'session' seems to get screwed up. For example my stored variables disappear, and I have to re-load my packages etc. It is difficult to give a reproducible example because the behaviour is inconsistent. I can run the same code twice, one time it is fine, the next it is not. Below I give a picture of it happening in my notebook, and also the driver logs for the time it occurred. Grateful if someone has an idea as to what is causing this? Thanks.

Atiq · ‎04-04-2022

Hey @Kaniz Fatma

We are also experiencing the same problem, and could not find a solution online. Could you kindly help us in this matter?

GC-James · ‎04-04-2022

@Atiquzzaman Ankur -- I think that this issue might be memory related. I think perhaps it happens when R tries to use more memory than is available to the Databricks cluster. Suggest monitoring the Ganglia UI just before you run a process that causes the crash. See if it 'tops out' the memory.

GC-James · ‎04-04-2022

@Kaniz Fatma -- I'm not sure if the issue persists in RStudio or not. It would take quite some effort to port my code out of databricks and test.

Atiq · ‎04-04-2022

@James Smith Thank you for the suggestion :). I will try that out.

@Kaniz Fatma And from our end, the script works perfectly fine on RStudio. Only getting the issue in Databricks.

GC-James · ‎04-04-2022

Well I have been limiting the memory usage of my code to 'solve' the problem so far. If it was the problem at all. I'm not sure.

GC-James · ‎04-27-2022

Hello @Kaniz Fatma . None of the above replies have given a solution.

data_warrior · ‎05-30-2022

Hello all @James Smith @Atiquzzaman Ankur @Kaniz Fatma ,

I am facing the exact same issue and do not know how to solve it. I tried freeing up some memory, But the problem persists. Please look at some of my screenshots.

It would be great if one of you could point me in the correct direction. I checked my memory consumption on driver node and I observed that it is maxing out for a short while. But I don't think this is the cause for failure as I have observed this multiple times before and the code still ran successfully without any error.

Best regards,

Akshay