topic Re: Databricks resets notebook all the time in Data Engineering

Databricks resets notebook all the time

jenshumrich — Fri, 09 Aug 2024 07:52:46 GMT

Whenever I run my script it resets the notebook state:
"The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached.

at com.databricks.spark.chauffeur.Chauffeur.onDriverStateChange(Chauffeur.scala:1467)"
This means I cannot even figure out efficiently what is causing the trouble. I am trying to do some data manipulation in R on the driver node. The amount of data is relatively small (a few GB) but I constantly run into this issue.

Re: Databricks resets notebook all the time

jenshumrich — Fri, 09 Aug 2024 07:56:26 GMT

It seems to have to do with some form of memory limit of the R session, but I cannot figure out:
1. Which parameter needs to be adjusted here
2. Why any sane person would setup an R session with only 2 GB of memory in the first place

Re: Databricks resets notebook all the time

jenshumrich — Fri, 09 Aug 2024 08:31:22 GMT

To add insult to injury it says this:

Re: Databricks resets notebook all the time

jenshumrich — Fri, 09 Aug 2024 09:28:23 GMT

Alternatively if I try to write it as RDS I get this error:

Error in saveRDS(get(object), file = file_path): error writing to connection

Re: Databricks resets notebook all the time

jenshumrich — Fri, 09 Aug 2024 10:15:05 GMT

To get closer to the error:

There is same mystical size limit.