Databricks resets notebook all the time

jenshumrich
Contributor

Whenever I run my script it resets the notebook state:
"The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached.

at com.databricks.spark.chauffeur.Chauffeur.onDriverStateChange(Chauffeur.scala:1467)"
This means I cannot even figure out efficiently what is causing the trouble. I am trying to do some data manipulation in R on the driver node. The amount of data is relatively small (a few GB) but I constantly run into this issue.

jenshumrich
Contributor

It seems to have to do with some form of memory limit of the R session, but I cannot figure out:
1. Which parameter needs to be adjusted here
2. Why any sane person would setup an R session with only 2 GB of memory in the first place

 

jenshumrich
Contributor

To add insult to injury it says this:

jenshumrich_0-1723191626374.png

 

jenshumrich
Contributor

Alternatively if I try to write it as RDS I get this error:

jenshumrich_0-1723195689156.png

Error in saveRDS(get(object), file = file_path): error writing to connection

jenshumrich
Contributor

To get closer to the error:

jenshumrich_0-1723198457103.png

There is same mystical size limit.