Combine Python + R in data manipulation in Databricks Notebook

Osky_Rosky
New Contributor II

Want to combine Py + R

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("CreateDataFrame").getOrCreate()

# Create a sample DataFrame

data = [("Alice", 25), ("Bob", 30), ("Charlie", 35), ("Oscar",36), ("Hiromi",41), ("Alejandro", 42)]

df = spark.createDataFrame(data, ["Name", "Age"])

display(df)

And R

%r

install.packages("sparklyr", version ="1.8.0")

library(sparklyr)

# Connect to the same Spark cluster

sc <- spark_connect(master = "yarn-client", version = "1.8.0"

)

But I have the error

**Error in spark_connect_gateway(gatewayAddress, gatewayPort, sessionId, : Gateway in localhost:8880 did not respond.

Try running 

options(sparklyr.log.console = TRUE)

 followed by 

sc <- spark_connect(...)

 for more debugging info. Some( Error in spark_connect_gateway(gatewayAddress, gatewayPort, sessionId, : Gateway in localhost:8880 did not respond. )**

Any Idea how can I combine both programming Languages in Databricks notebook?