Unable to convert R dataframe to spark dataframe
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-20-2025 02:17 PM
Hi All,
Does anyone knows how to convert R dataframe to spark dataframe to Pandas dataframe? I wanted to get a Pandas dataframe ultimately but I guess I need to convert to spark first. I've been using this sparklyr library but my code did not work. This is the code I used in my R cell:
%r
library(sparklyr)
library(SparkR)
sc = spark_connect(method = "databricks")
matched_rdf = psm_tbl %>% select(c(code_treat, code_control)) %>% data.frame()
# write the R dataframe to spark
matched_spark = copy_to(sc, matched_rdf, overwrite = TRUE)
I suppose matched_spark is spark dataframe already and on the next cell I write:
select * from matched_spark, but there's an error saids "matched_spark" object not found.
Appreciate if anyone could help!
