User16752239289
Databricks Employee
Databricks Employee

This is due to the when users run their R scripts on Rstudio, the R session is not shut down gracefully.

Databricks is working on handle the R session better and removed the limit.

As a workaround, you can create and run below init script to increase the limit:

%scala
val initScriptContent = s"""
 |#!/bin/bash
 |cat > /databricks/common/conf/rbackend_limit.conf << EOL
 |{
 | databricks.daemon.driver.maxNumRBackendsPerDriver = <value>
 |}
 |EOL
""".stripMargin
dbutils.fs.put("dbfs:/<path>/set_rbackend.sh",initScriptContent, true)

View solution in original post