Databricks resets notebook all the time

[jenshumrich]

Whenever I run my script it resets the notebook state:
"The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached.

at com.databricks.spark.chauffeur.Chauffeur.onDriverStateChange(Chauffeur.scala:1467)"
This means I cannot even figure out efficiently what is causing the trouble. I am trying to do some data manipulation in R on the driver node. The amount of data is relatively small (a few GB) but I constantly run into this issue.

[jenshumrich]

It seems to have to do with some form of memory limit of the R session, but I cannot figure out:
1. Which parameter needs to be adjusted here
2. Why any sane person would setup an R session with only 2 GB of memory in the first place

[jenshumrich]

To add insult to injury it says this:

[jenshumrich]

Alternatively if I try to write it as RDS I get this error:

Error in saveRDS(get(object), file = file_path): error writing to connection

[jenshumrich]

To get closer to the error:

There is same mystical size limit.