cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Spark Bigquery Connector Availability

JaviPA
New Contributor

Hello,
We are trying to use this library (https://github.com/GoogleCloudDataproc/spark-bigquery-connector) to read Bigquery data from a Databricks cluster in Azure, could someone confirm if this library is fully available and supported on Databricks? From what we have reviewed, there is no problem in Google Cloud Dataproc but we would like to know if it is the same for Databricks.
In this link (https://docs.databricks.com/en/connect/external-systems/bigquery.html) it is indicated that the configuration is experimental, but it is not clear to me if it refers to the library itself or to the federated connections to Bigquery.

 

1 REPLY 1

szymon_dybczak
Contributor III

Hi @JaviPA ,

Documentation referes to the library you're going to use:

GitHub - GoogleCloudDataproc/spark-bigquery-connector: BigQuery data source for Apache Spark: Read d...

Also, it's mentioned that if you want to have full query federation support, you should use Lakehouse Federation:

Run federated queries on Google BigQuery | Databricks on AWS

Slash_0-1724839845372.png

But if in your case library does the job, then just use it 🙂

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group