topic Why are R scripts inside of Databricks notebooks creating writeLines errors? in Data Engineering

Why are R scripts inside of Databricks notebooks creating writeLines errors?

DoD — Wed, 19 Jan 2022 03:42:43 GMT

I recently posted this in Stack Overflow. I'm using R in Databricks. R Studio runs fine and executes from the Databricks cluster. I would like to transition from R Studio to notebooks. When I start the cluster, R seems to run fine from notebooks. I would like to run an Rmarkdown file. When I render the Rmarkdown file, it seems to run fine but throws the error below when it is done. If I run the Rmarkdown again, it throws the error immediately. R will not run after throwing this error once - it just repeats, even for the simplest of scripts. Any help is greatly appreciated!

Error in writeLines(DB_STREAM_BEACON_STRING_START__, DATABRICKS_STDERRCONNSTREAM__) : 
  invalid connection
Some(<code style = 'font-size:10p'>  </code>)

Re: Why are R scripts inside of Databricks notebooks creating writeLines errors?

DoD — Wed, 19 Jan 2022 15:51:13 GMT

Figured it out after several hours of research and troubleshooting. This problem arises from Rmarkdown failing to write various types of common Rmarkdown output (most likely viewer output) to stdout. Rather than directly running

rmarkdown::render()

from the notebook, run

R -e 'rmarkdown::render()' > out.txt

as a shell script in the notebook, redirecting output to a text file.

Re: Why are R scripts inside of Databricks notebooks creating writeLines errors?

Anonymous — Wed, 19 Jan 2022 16:15:56 GMT

@Paul Evangelista - Thank you for letting us know. You did great!

Would you be happy to mark your answer as best so that others can find your solution more easily?