Spark Bigquery Connector Availability

JaviPA — Wed, 28 Aug 2024 09:30:06 GMT

Hello,
We are trying to use this library (https://github.com/GoogleCloudDataproc/spark-bigquery-connector) to read Bigquery data from a Databricks cluster in Azure, could someone confirm if this library is fully available and supported on Databricks? From what we have reviewed, there is no problem in Google Cloud Dataproc but we would like to know if it is the same for Databricks.
In this link (https://docs.databricks.com/en/connect/external-systems/bigquery.html) it is indicated that the configuration is experimental, but it is not clear to me if it refers to the library itself or to the federated connections to Bigquery.

Re: Spark Bigquery Connector Availability

szymon_dybczak — Wed, 28 Aug 2024 10:12:13 GMT

Hi @JaviPA ,

Documentation referes to the library you're going to use:

GitHub - GoogleCloudDataproc/spark-bigquery-connector: BigQuery data source for Apache Spark: Read data from BigQuery into DataFrames, write DataFrames into BigQuery tables.

Also, it's mentioned that if you want to have full query federation support, you should use Lakehouse Federation:

Run federated queries on Google BigQuery | Databricks on AWS

But if in your case library does the job, then just use it 🙂

topic Spark Bigquery Connector Availability in Data Engineering

Spark Bigquery Connector Availability

Re: Spark Bigquery Connector Availability