Load GCP data to Databricks using R

dZegpi
New Contributor II

I'm working with Databricks and Google Cloud in the same project. I want to load specific datasets stored in GCP into a R notebook in Databricks. I currently can see the datasets in BigQuery. The problem is that using the sparklyr package, I'm not able to see the datasets that I have in GCP. Instead, I see the name of other datasets I was not aware of. 

This is my R code

 

library(sparklyr)
sc <- spark_connect(method = "databricks")

# List table names in my spark connection
dplyr::src_tbls(sc)
# Can't see the tables sotred in GCP

# Try (and fail) to load the tables
spark_read_table(sc, "my-table-name")

 

How can I access my tables stored in GCP through Databricks notebooks using R?; Is sparklyr the right approach?

dZegpi
New Contributor II

Other than library(SparkR), this is the same code I posted. This does not solve my problem.