โ02-23-2022 04:27 AM
Sometimes when I am running R code in a Databricks notebook I am given this error. The cell I am running fails, and my whole R 'session' seems to get screwed up. For example my stored variables disappear, and I have to re-load my packages etc. It is difficult to give a reproducible example because the behaviour is inconsistent. I can run the same code twice, one time it is fine, the next it is not. Below I give a picture of it happening in my notebook, and also the driver logs for the time it occurred. Grateful if someone has an idea as to what is causing this? Thanks.
โ02-23-2022 08:52 AM
Hi @James Smithโ ! Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will get back to you soon. Thanks.
โ03-09-2022 04:14 AM
Hi @James Smithโ , Does this issue remain the same in RStudio or is this error specific to only the Databricks Notebook?
โ04-04-2022 01:50 AM
Hey @Kaniz Fatmaโ
We are also experiencing the same problem, and could not find a solution online. Could you kindly help us in this matter?
โ04-04-2022 02:21 AM
@Atiquzzaman Ankurโ -- I think that this issue might be memory related. I think perhaps it happens when R tries to use more memory than is available to the Databricks cluster. Suggest monitoring the Ganglia UI just before you run a process that causes the crash. See if it 'tops out' the memory.
โ04-04-2022 02:20 AM
@Kaniz Fatmaโ -- I'm not sure if the issue persists in RStudio or not. It would take quite some effort to port my code out of databricks and test.
โ04-04-2022 02:40 AM
Hi @James Smithโ , Were you able to resolve your problem? Do you still need help?
โ04-04-2022 02:58 AM
@James Smithโ Thank you for the suggestion :). I will try that out.
@Kaniz Fatmaโ And from our end, the script works perfectly fine on RStudio. Only getting the issue in Databricks.
โ04-04-2022 02:59 AM
Well I have been limiting the memory usage of my code to 'solve' the problem so far. If it was the problem at all. I'm not sure.
โ04-27-2022 01:50 AM
Hi @James Smithโ , Just a friendly follow-up. Do you still need help, or do the above responses help you find the solution? Please let us know.
โ04-27-2022 02:47 AM
Hello @Kaniz Fatmaโ . None of the above replies have given a solution.
โ05-30-2022 09:44 AM
Hello all @James Smithโ @Atiquzzaman Ankurโ @Kaniz Fatmaโ ,
I am facing the exact same issue and do not know how to solve it. I tried freeing up some memory, But the problem persists. Please look at some of my screenshots.
It would be great if one of you could point me in the correct direction. I checked my memory consumption on driver node and I observed that it is maxing out for a short while. But I don't think this is the cause for failure as I have observed this multiple times before and the code still ran successfully without any error.
Best regards,
Akshay
โ05-30-2022 09:44 AM
The error file is attached here.