topic Re: sparkR.session in Get Started Discussions

sparkR.session

Flachboard84 — Tue, 27 Feb 2024 16:35:01 GMT

Why might this be erroring out? My understanding is that SparkR is built into Databricks.

Code:

library(SparkR, include.only=c('read.parquet', 'collect'))

sparkR.session()

Error:

Error in sparkR.session(): could not find function "sparkR.session"

shan_chandra — Tue, 27 Feb 2024 16:46:08 GMT

@Flachboard84 - could you please try the below code ?

library(SparkR, include.only=c('read.parquet', 'collect'))

sparkR.session() is not required on the databricks notebook. This should be handled automatically.

Flachboard84 — Tue, 27 Feb 2024 17:03:40 GMT

Thanks for the reply!

Now I'm getting this:

Warning in sparkR.callJMethod(SparkR:::getSparkContext(), "setLocalProperty", : could not find function "sparkR.callJMethod"

shan_chandra — Tue, 27 Feb 2024 17:25:39 GMT

@Flachboard84 - could you please share the associated code?

Flachboard84 — Tue, 27 Feb 2024 17:33:00 GMT

It happens with any code; even something as simple as...

x <- 2 + 2