Hi @soumiknow,

To resolve the 'connection refused' error when using the google-cloud-bigquery library in your Databricks notebook, you need to ensure that your Databricks cluster is properly configured to authenticate with Google Cloud Platform (GCP). Here are the steps you can follow:

1. Create a Google Service Account and Key:
• Go to the Google Cloud Console.
• Navigate to IAM & Admin > Service Accounts.
• Click Create Service Account.
• Provide a name and description for the service account.
• Assign the necessary roles (e.g., BigQuery Data Viewer, BigQuery Job User).
• Click Create and then Done.
• In the Service Accounts list, find your new service account and click on it.
• Go to the Keys section, click Add Key, and select Create new key.
• Choose JSON and click Create. This will download a JSON key file to your computer.
2. Upload the JSON Key File to Databricks:
• In your Databricks workspace, go to Data > DBFS > Upload and upload the JSON key file.
3. Configure the Databricks Cluster:
• Go to Clusters and select your cluster.
• Click on the Spark Config tab.
• Add the following Spark configuration, replacing `<path-to-json-key>` with the path to your uploaded JSON key file:

spark.hadoop.google.cloud.auth.service.account.enable true

spark.hadoop.google.cloud.auth.service.account.json.keyfile /dbfs/<path-to-json-key>

Set the Environment Variable:

• In your notebook, set the GOOGLE_APPLICATION_CREDENTIALS environment variable to point to the JSON key file:

import os

os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/dbfs/<path-to-json-key>"

Run Your BigQuery Code:

• Now you can run your BigQuery code in the notebook. Here is an example:

from google.cloud import bigquery

gcp_project_id = "gcp-test"

drop_partition_query = f""" DELETE FROM `gcp-test.test-dataset.test-table` WHERE partition_date = '2022-01-22' """

print(drop_partition_query)

client = bigquery.Client(project=gcp_project_id)

result = client.query(drop_partition_query)

Please review this documentation: https://docs.databricks.com/en/connect/external-systems/bigquery.html