Hi @Parul Paul​ , could you please check if this is the scenario: https://stackoverflow.com/questions/68623803/load-to-bigquery-via-spark-job-fails-with-an-exception-...

Also, you can refer: https://github.com/GoogleCloudDataproc/spark-bigquery-connector/issues/55

using big query connector with spark , please refer: https://cloud.google.com/dataproc/docs/tutorials/bigquery-connector-spark-example

Also, the issue can be: multiple versions of spark sql in the class path, which could be because of libraries installed on the cluster. Could you please check uninstalling few of the unnecessary libraries and restarting the cluster?