Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
If I start a RStudio Server with in cluster init script as described here in a Unity Catalog Cluster the sparklyr connection fails with an error about a missing Credential Scope.=LI tried it both in 11.3LTS and 12.0 Beta. I tried it only in a Persona...
When installing Notebook-scoped R libraries I don't want to manually specify the custom CRAN mirror each time like this:install.packages("diffdf", repos="my_custom_cran_url'')Instead I want to take the custom CRAN mirror URL by default so that I don'...
Attempting to install SparkR to the cluster and successfully installed other packages such as tidyverse via CRAN. The error is copied below, any help you can provide is greatly appreciated!Databricks runtime 10.4 LTSLibrary installation attempted on ...
Hi @Ross Hamilton ,I believe SparkR comes inbuilt with Databricks RStudio and you don't have to install it explicitly. You can directly import it with library(SparkR) and it works for you from your above comment.The error message you see could be re...
I'm operating on some data that looks like the image attached.
the command that I'm performing is :<code>library(magrittr)
subsetting the data for MAC-OS & sorting by event-timestamp.
acDF <- eventsDF %>% SparkR::select("device", "event_timestamp...
When run sparkR.session()I faced below error:Spark package found in SPARK_HOME: /databricks/spark
Launching java with spark-submit command /databricks/spark/bin/spark-submit sparkr-shell /tmp/Rtmp5hnW8G/backend_porte9141208532d
Error: Could not f...
This is due to the when users run their R scripts on Rstudio, the R session is not shut down gracefully. Databricks is working on handle the R session better and removed the limit. As a workaround, you can create and run below init script to increase...
I have the following sparkdataframe :
sale_id/ created_at
1 /2016-05-28T05:53:31.042Z
2 /2016-05-30T12:50:58.184Z
3/ 2016-05-23T10:22:18.858Z
4 /2016-05-27T09:20:15.158Z
5 /2016-05-21T08:30:17.337Z
6 /2016-05-28T07:41:14.361Z
i need t add a year-wee...
