Kumaran
Databricks Employee
Databricks Employee

Hi @JefferyReichman,

When trying to read a Hive table through Sparklyr, you can use the spark_read_table() function. This function reads tables from your cluster's default database or a specific database.

Here's an example of how to read a Hive table in Sparklyr using a specific database:

 

%r
# Load Sparklyr library
library(sparklyr)

# Connect to the cluster using a service principal
sc <- spark_connect(method = "databricks", 
                    username = "client_id",
                    password = "client_secret",
                    tenant_id = "tenant_id",
                    endpoint = "https://westus2.azuredatabricks.net")

# Set the database where the table is located
database_name <- "xxx_mydata"

# Use spark_read_table() function to read the table
data_tbl <- spark_read_table(sc, in_database(database_name, "mydata_etl"))