SparkR session failed to initialize

User16752239289 — Fri, 02 Jul 2021 16:11:29 GMT

When run sparkR.session()

I faced below error:

Spark package found in SPARK_HOME: /databricks/spark
 
Launching java with spark-submit command /databricks/spark/bin/spark-submit sparkr-shell /tmp/Rtmp5hnW8G/backend_porte9141208532d
 
Error: Could not find or load main class org.apache.spark.launcher.Main
 
/databricks/spark/bin/spark-class: line 101: CMD: bad array subscript
 
Error in sparkR.sparkContext(master, appName, sparkHome, sparkConfigMap, :
 
JVM is not ready after 10 seconds

When I checked the cluster log4j , I found I hit the Rbackend limit:

21/06/29 18:26:17 INFO RDriverLocal: 394. RDriverLocal.e9dee079-46f8-4108-b1ed-25fa02742efb: Exceeded maximum number of RBackends limit: 200

Re: SparkR session failed to initialize

User16752239289 — Fri, 02 Jul 2021 16:18:23 GMT

This is due to the when users run their R scripts on Rstudio, the R session is not shut down gracefully.

Databricks is working on handle the R session better and removed the limit.

As a workaround, you can create and run below init script to increase the limit:

%scala
val initScriptContent = s"""
 |#!/bin/bash
 |cat > /databricks/common/conf/rbackend_limit.conf << EOL
 |{
 | databricks.daemon.driver.maxNumRBackendsPerDriver = <value>
 |}
 |EOL
""".stripMargin
dbutils.fs.put("dbfs:/<path>/set_rbackend.sh",initScriptContent, true)

topic SparkR session failed to initialize in Data Engineering

SparkR session failed to initialize

Re: SparkR session failed to initialize