Spark Bigquery Connector Availability
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-28-2024 02:30 AM
Hello,
We are trying to use this library (https://github.com/GoogleCloudDataproc/spark-bigquery-connector) to read Bigquery data from a Databricks cluster in Azure, could someone confirm if this library is fully available and supported on Databricks? From what we have reviewed, there is no problem in Google Cloud Dataproc but we would like to know if it is the same for Databricks.
In this link (https://docs.databricks.com/en/connect/external-systems/bigquery.html) it is indicated that the configuration is experimental, but it is not clear to me if it refers to the library itself or to the federated connections to Bigquery.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-28-2024 03:12 AM
Hi @JaviPA ,
Documentation referes to the library you're going to use:
GitHub - GoogleCloudDataproc/spark-bigquery-connector: BigQuery data source for Apache Spark: Read d...
Also, it's mentioned that if you want to have full query federation support, you should use Lakehouse Federation:
Run federated queries on Google BigQuery | Databricks on AWS
But if in your case library does the job, then just use it 🙂