โ08-20-2023 07:35 PM
Databricks Community
New to Databricks, and R User and trying to figure out how to load a hive table via Sparklyr. The path to the file is https://databricks.xxx.xx.gov/#table/xxx_mydata/mydata_etl (right clicking on the file). I tried
data_tbl <- tbl(sc, "https://databricks.xxx.xx.gov/#table/xxx_mydata/mydata_etl") and apparently that isn't correct.
Jeff
โ08-23-2023 05:31 AM
Those set of commands didn't seem to work. However, with a little digging and reading I found this set of command did work.
%r
# Load Sparklyr library
library(sparklyr)
# Connect to the cluster using a service principal
sc <- spark_connect(method = "databricks")
# Set the database where the table is located
tbl_change_db <- "xxx_mydata"
# Use spark_read_table() function to read the table
data_tbl <- spark_read_table(sc, "mydata_etl")
โ08-20-2023 11:51 PM
Hi @JefferyReichman,
When trying to read a Hive table through Sparklyr, you can use the spark_read_table() function. This function reads tables from your cluster's default database or a specific database.
Here's an example of how to read a Hive table in Sparklyr using a specific database:
%r
# Load Sparklyr library
library(sparklyr)
# Connect to the cluster using a service principal
sc <- spark_connect(method = "databricks",
username = "client_id",
password = "client_secret",
tenant_id = "tenant_id",
endpoint = "https://westus2.azuredatabricks.net")
# Set the database where the table is located
database_name <- "xxx_mydata"
# Use spark_read_table() function to read the table
data_tbl <- spark_read_table(sc, in_database(database_name, "mydata_etl"))
โ08-23-2023 05:31 AM
Those set of commands didn't seem to work. However, with a little digging and reading I found this set of command did work.
%r
# Load Sparklyr library
library(sparklyr)
# Connect to the cluster using a service principal
sc <- spark_connect(method = "databricks")
# Set the database where the table is located
tbl_change_db <- "xxx_mydata"
# Use spark_read_table() function to read the table
data_tbl <- spark_read_table(sc, "mydata_etl")
โ08-22-2023 02:53 PM
Thanks - where can I read up on this for getting started - Jeff
โ08-23-2023 09:55 AM
Hi @JefferyReichman,
Not sure that I completely understood your last question about "where I can read up on this for getting started". However, you can start by running this code in the Databricks community edition notebook.
For more details: Link
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group