I want to use google-cloud-bigquery library in my PySpark code though I know that spark-bigquery-connector is available. The reason I want to use is that the Databricks Cluster 15.4LTS comes with 0.22.2-SNAPSHOT version of spark-bigquery-connector which is not supposed to be a working version as per their documentation. They suggested to use 0.41.0 version of spark-bigquery-connector jar in the databricks cluster to leverage all the features.

Since I am unable to use 0.41.0 jar in the Databricks Cluster 15.4LTS, I want to use google-cloud-bigquery lib.

The problem I facing with the below code is:

from google.cloud import bigquery

gcp_project_id = "gcp-test"
drop_partition_query = f""" DELETE FROM `gcp-test.test-dataset.test-table` WHERE partition_date = '2022-01-22' """ 
print(drop_partition_query)
client = bigquery.Client(project=f"{gcp_project_id}")
result = client.query(drop_partition_query)

WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 5 of 5. Reason: [Errno 111] Connection refused
WARNING:google.auth._default:No project ID could be determined. Consider running `gcloud config set project` or setting the GOOGLE_CLOUD_PROJECT environment variable
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 1 of 5. Reason: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/universe/universe_domain (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7b322f6d1b10>: Failed to establish a new connection: [Errno 111] Connection refused'))

I tried to set the GOOGLE_CLOUD_PROJECT as shown below:

%env GOOGLE_CLOUD_PROJECT=gcp-test

But still getting the same 'connection refused' error like below (only the project ID warning got resolved)

WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 5 of 5. Reason: [Errno 111] Connection refused
WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 1 of 5. Reason: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/universe/universe_domain (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7b322f6d1b10>: Failed to establish a new connection: [Errno 111] Connection refused'))

Can anyone suggest me how can I use a google-cloud library in Databricks notebook or how can I resolved this 'connection refused' error in Databricks notebook.

#Bigquery #GoogleCloud