@Oscar CENTENO MORA​ :

To combine Py and R in a Databricks notebook, you can use the magics command %python and %r

to switch between Python and R cells. Here's an example of how to create a Spark DataFrame in Python and then use it in R:

from pyspark.sql import SparkSession
 
# Create a Spark session
spark = SparkSession.builder.appName("CreateDataFrame").getOrCreate()
 
# Create a sample DataFrame in Python
data = [("Alice", 25), ("Bob", 30), ("Charlie", 35), ("Oscar",36), ("Hiromi",41), ("Alejandro", 42)]
df = spark.createDataFrame(data, ["Name", "Age"])
 
# Use the %python magic to switch to a Python cell
%python
 
# Convert the Python DataFrame to an R DataFrame using sparklyr
library(sparklyr)
library(dplyr)
sdf <- spark_dataframe(df)
rdf <- sdf %>% invoke("toDF", "Name", "Age")
 
# Use the %r magic to switch to an R cell
%r
 
# Print the R DataFrame
print(rdf)

Note that the sparklyr package must be installed in the R environment using the install.packages()

function, as shown in your example. Also, make sure that the Spark cluster is running and accessible from your notebook.