SQL table convert to R dataframe

tt_mmo · ‎01-08-2025

I have a table with ~6 million rows. I am attempting to convert this from a sql table on my catalog to an R dataframe to use the tableone package. I separate my table into 3 tables each containing about 2 million rows then ran it through tbl() and as.data.frame(). Finally merged the 3 tables back together. This worked, but is there a way to perform this without the subsetting my original 6 million row dataset? Been troubleshoot for about 1 week now. Thanks

Code below:

%r

# Install packages if not already installed

if (!require("sparklyr")) install.packages("sparklyr")

if (!require("dplyr")) install.packages("dplyr")

library(sparklyr)

library(dplyr)

#configure

config <- spark_config()

config$spark.driver.memory <- "32G"

config$spark.executor.memory <- "32G"

config$spark.memory.fraction <- 0.8

#connect to databricks

sc = spark_connect(method = 'databricks', version = '2.0.0', config = config)

# Copy table from SQL to R object with tables already separated out

m1 <- tbl(sc, "final_lab_mstr_3_1")

m2 <- tbl(sc, "final_lab_mstr_3_2")

m3 <- tbl(sc, "final_lab_mstr_3_3")

# Convert Spark DataFrame to R DataFrame

df_1 <- as.data.frame(collect(m1))

df_2 <- as.data.frame(collect(m2))

df_3 <- as.data.frame(collect(m3))

# Merge the data frames

df3 <- rbind(df_1, df_2, df_3)