Hello @Alberto_Umana & @VZLA ,

Thanks both of you for your kind suggestions.

I was able to query BQ using spark-bigquery-connector with 0.22.2-SNAPSHOT version. The problem started when I tried to use 'spark.sql.sources.partitionOverwriteMode' to 'DYNAMIC'. I already explained the issue in the community post. Even though removing the 0.22.2-SNAPSHOT version by init_script was working, but the expected version 0.41.0 was not working after installing. So I choose google-cloud-bigquery lib to perform the partitioned data overwrite operation.

I am now able to resolve the 'connection refused' error by choosing 'Access Mode' to 'No isolation shared' while creating the cluster. Now each BQ query is returning me the expected output through google-cloud-bigquery client. Let me know if there is any concern over choosing 'No isolation shared' access mode.