Not, the init scripts run before Spark starts or any packages get loaded. So if there are any dependancies, they will need to be stated. Also, I think if the user does install,packages("your_library"), then Databricks will automatically install on all nodes. Also installing using the library UI will do this. But we're just making solutions based on hypotheses here. We would really need to know what error you are seeing to tell.

Typically- whatever R library you are installing on the cluster should ALSO install the JAR files. My guess is that R's arrow package does this, but not sure. It definitely installs the underlying C++ dependancy. Not sure if there is also a Java component.