topic Re: Failed to install cluster scoped SparkR library in Data Engineering

Failed to install cluster scoped SparkR library

Ross — Thu, 04 Aug 2022 12:10:15 GMT

Attempting to install SparkR to the cluster and successfully installed other packages such as tidyverse via CRAN. The error is copied below, any help you can provide is greatly appreciated!

Databricks runtime 10.4 LTS

Library installation attempted on the driver node of cluster 0527-101647-p8flqbiy and failed. Please refer to the following error message to fix the library or contact Databricks support. Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: Error: Error installing R package: Could not install package with error: package ‘SparkR’ is not available for this version of R

Full error log available at /databricks/driver/library-install-logs/r-package-installation-2022-08-04T12:04:30Z-8gumapnw.log

Error: Error installing R package: Could not install package with error: package ‘SparkR’ is not available for this version of R

Full error log available at /databricks/driver/library-install-logs/r-package-installation-2022-08-04T12:04:30Z-8vp_gpk1.log

Re: Failed to install cluster scoped SparkR library

Prabakar — Thu, 04 Aug 2022 12:23:10 GMT

hi @Ross Hamilton this looks like the packages are failing to get installed. You can check the log to understand what is the issue.

I believe the failure is caused because of dependency missing. I would recommend checking for the dependent libraries and installing them.

You can try the below steps as well.

a) Install lib using command from notebook.

install.packages()

b) Copy installed lib to dbfs.

cp -R /local_disk/env /dbfs/path_to_r_library

c) Use init script to get installed libs in cluster lib path.

## Define contents of script
script = """
#!/bin/bash
R --vanilla <<EOF
system("cp -R /dbfs/path_to_r_library /databricks/spark/R/lib/", intern = T)
q()
EOF
"""

Re: Failed to install cluster scoped SparkR library

Ross — Thu, 04 Aug 2022 13:11:53 GMT

I'm afraid the solution doesn't work:

install.packages("SparkR")
 
Installing package into ‘/local_disk0/.ephemeral_nfs/envs/rEnv-1c6d7759-7751-4681-8489-a027452405f0’
(as ‘lib’ is unspecified)
Warning: package ‘SparkR’ is not available for this version of R
 
A version of this package for your version of R might be available elsewhere,
see the ideas at
https://cran.r-project.org/doc/manuals/r-patched/R-admin.html#Installing-packages

However if I go to load the library SparkR this works fine. I think it's the same issue when installing the package to the cluster.

library(SparkR)
 
Attaching package: ‘SparkR’
 
The following object is masked _by_ ‘.GlobalEnv’:
 
    setLocalProperty
 
The following objects are masked from ‘package:stats’:
 
    cov, filter, lag, na.omit, predict, sd, var, window
 
The following objects are masked from ‘package:base’....

Re: Failed to install cluster scoped SparkR library

Prabakar — Thu, 04 Aug 2022 14:24:08 GMT

Do you find any information in the logs?

Full error log available at /databricks/driver/library-install-logs/r-package-installation-2022-08-04T12:04:30Z-8vp_gpk1.log

Re: Failed to install cluster scoped SparkR library

Vivian_Wilfred — Thu, 04 Aug 2022 21:08:54 GMT

Hi @Ross Hamilton ,

I believe SparkR comes inbuilt with Databricks RStudio and you don't have to install it explicitly. You can directly import it with library(SparkR) and it works for you from your above comment.

The error message you see could be red herring. Does this answer your question or am I missing something?