AnalysisException: Multiple sources found for bigquery (com.google.cloud.spark.bigquery.BigQueryRelationProvider, com.google.cloud.spark.bigquery.v2.BigQueryTableProvider), please specify the fully qualified class name.

parulpaul
New Contributor III

‎10-18-2022 12:53 AM

While reading data from BigQuery to Databricks

getting the error : AnalysisException:

Multiple sources found for bigquery (com.google.cloud.spark.bigquery.BigQueryRelationProvider, com.google.cloud.spark.bigquery.v2.BigQueryTableProvider),

please specify the fully qualified class name.

Jar used - spark-bigquery-with-dependencies_2.12-0.27.0.jar

df = spark.read.format("bigquery")\

        .option("credentialsFile",credientialFilePath)\

        .option("ParentProject",projectName)\

        .option("table",tableName)\

        .load()

Debayan
Databricks Employee
Databricks Employee

‎10-18-2022 08:14 AM

Hi @Parul Paul​ , could you please check if this is the scenario: https://stackoverflow.com/questions/68623803/load-to-bigquery-via-spark-job-fails-with-an-exception-...

Also, you can refer: https://github.com/GoogleCloudDataproc/spark-bigquery-connector/issues/55

using big query connector with spark , please refer: https://cloud.google.com/dataproc/docs/tutorials/bigquery-connector-spark-example

Also, the issue can be: multiple versions of spark sql in the class path, which could be because of libraries installed on the cluster. Could you please check uninstalling few of the unnecessary libraries and restarting the cluster?