topic Re: AnalysisException: Multiple sources found for bigquery (com.google.cloud.spark.bigquery.BigQueryRelationProvider, com.google.cloud.spark.bigquery.v2.BigQueryTableProvider), please specify the fully qualified class name. in Data Engineering

AnalysisException: Multiple sources found for bigquery (com.google.cloud.spark.bigquery.BigQueryRelationProvider, com.google.cloud.spark.bigquery.v2.BigQueryTableProvider), please specify the fully qualified class name.

parulpaul — Tue, 18 Oct 2022 07:53:53 GMT

While reading data from BigQuery to Databricks

getting the error : AnalysisException:

Multiple sources found for bigquery (com.google.cloud.spark.bigquery.BigQueryRelationProvider, com.google.cloud.spark.bigquery.v2.BigQueryTableProvider),

please specify the fully qualified class name.

Jar used - spark-bigquery-with-dependencies_2.12-0.27.0.jar

df = spark.read.format("bigquery")\

.option("credentialsFile",credientialFilePath)\

.option("ParentProject",projectName)\

.option("table",tableName)\

.load()

Re: AnalysisException: Multiple sources found for bigquery (com.google.cloud.spark.bigquery.BigQueryRelationProvider, com.google.cloud.spark.bigquery.v2.BigQueryTableProvider), please specify the fully qualified class name.

Debayan — Tue, 18 Oct 2022 15:14:54 GMT

Hi @Parul Paul , could you please check if this is the scenario: https://stackoverflow.com/questions/68623803/load-to-bigquery-via-spark-job-fails-with-an-exception-for-multiple-sources-foun

Also, you can refer: https://github.com/GoogleCloudDataproc/spark-bigquery-connector/issues/55

using big query connector with spark , please refer: https://cloud.google.com/dataproc/docs/tutorials/bigquery-connector-spark-example

Also, the issue can be: multiple versions of spark sql in the class path, which could be because of libraries installed on the cluster. Could you please check uninstalling few of the unnecessary libraries and restarting the cluster?