@soumiknow
- This is not the output from -verbose:class, what you see is likely coming from importing the library from an external repository and its showing the add dependencies process. Telling it has pulled and downloaded the "com.google.cloud.spark#spark-bigquery-connector-common;0.41.0!spark-bigquery-connector-common.jar", which is probably added via a sparkSession or Cluster Library with Maven as source. The verbose:class flag, prints in the STDOUT output, something like this(classloading):
- [Loaded com.google.cloud.spark.bigquery.BigQueryRelationProvider from file:/databricks/jars/----ws_3_5--third_party--bigquery-connector--spark-bigquery-connector-hive-2.3__hadoop-3.2_2.12--118181791--fatJar-assembly-0.22.2-SNAPSHOT.jar]
- In DBR 15.4 LTS, which you're already using, you also have already available the:
- ----ws_3_5--third_party--bigquery-connector--spark-bigquery-connector-hive-2.3__hadoop-3.2_2.12--118181791--fatJar-assembly-0.22.2-SNAPSHOT.jar, and
- ----ws_3_5--third_party--bigquery-connector--spark-bigquery-connector-upgrade_scala-2.12--118181791--spark-bigquery-with-dependencies_2.12-0.41.0.jar
- The answer you got from the Google Support team, is referring to the fatJar-assembly-0.22.2-SNAPSHOT. But the spark-bigquery v0.22, does have the "com.google.cloud.spark.bigquery.BigQueryRelationProvider" available as you can see in here https://github.com/GoogleCloudDataproc/spark-bigquery-connector/blob/branch-0.22/connector/src/main/... So the problem you're running into, is not related to the jar file version itself.
So at this point you have two different issues, 1) Need to use the v0.40, and 2) Currently getting a DATA_SOURCE_NOT_FOUND error.
- If you encounter the DATA_SOURCE_NOT_FOUND error, it means the data source name provided to Spark is not resolvable either in its built-in registry or through any dynamically loaded libraries. I'm honestly unsure how are you running into this error, and would need your clarification comments about the current cluster status and setup to help you with it. If I had to guess, I would say you have manually deleted this fat jar v0.20 from the cluster altogheter, using an init script maybe?
- When you call "format("bigquery")" this is what will happen behind the scenes in the DataSource.scala:
case name if name.equalsIgnoreCase("bigquery") =>
"com.google.cloud.spark.bigquery.BigQueryRelationProvider"
- So lookupDataSource will try to find the "com.google.cloud.spark.bigquery.BigQueryRelationProvider" and then Spark instantiates the "BigQueryRelationProvider". In other words, when you use spark.read.format("bigquery"), Spark uses this mapping to locate and load the appropriate class.
Could you please run this from a notebook in DBR 15.4 LTS with no additional libraries attached to it:
%python
class_name = "com.google.cloud.spark.bigquery.BigQueryRelationProvider"
try:
# Get the class reference
cls = spark._jvm.Thread.currentThread().getContextClassLoader().loadClass(class_name)
# Get the JAR file path
jar_path = cls.getProtectionDomain().getCodeSource().getLocation().getPath()
print(f"The class {class_name} is loaded from: {jar_path}")
except Exception as e:
print(f"Error locating the class {class_name}: {e}")
It should return:
The class com.google.cloud.spark.bigquery.BigQueryRelationProvider is loaded from: /databricks/jars/----ws_3_5--third_party--bigquery-connector--spark-bigquery-connector-hive-2.3__hadoop-3.2_2.12--118181791--fatJar-assembly-0.22.2-SNAPSHOT.jar
If it does not then, the reason clearly is the absence of a jar file with the class mentioned in the error message.
The MVN output you've shared though is actually interesting, solely based on my personal assumption, you've attempted to add the spark-bigquery artifact through Cluster Library with Maven as source, but the artifact that you have pulled does not have the BigQueryRelationProvider as well, if you're ok with going with the latest version, the Maven coordinates you should be using are "com.google.cloud.spark:spark-bigquery_2.12:0.41.1"
Then rerunning the same code:
class_name = "com.google.cloud.spark.bigquery.BigQueryRelationProvider"
try:
cls = spark._jvm.Thread.currentThread().getContextClassLoader().loadClass(class_name)
jar_path = cls.getProtectionDomain().getCodeSource().getLocation().getPath()
print(f"The class {class_name} is loaded from: {jar_path}")
# Print the package information (often contains version info)
package = cls.getPackage()
print(f"Package Specification Title: {package.getSpecificationTitle()}")
print(f"Package Specification Version: {package.getSpecificationVersion()}")
print(f"Package Implementation Version: {package.getImplementationVersion()}")
except Exception as e:
print(f"Error: {e}")
Should return:
The class com.google.cloud.spark.bigquery.BigQueryRelationProvider is loaded from: /local_disk0/tmp/addedFile235d6f3b981a4f61bb72e599c2d013986386268607538077001/com_google_cloud_spark_spark_bigquery_2_12_0_41_1.jar
Package Specification Title: BigQuery DataSource v1 for Scala 2.12
Package Specification Version: 0.41
Package Implementation Version: 0.41.1
Whether the library and your current cluster status after the changes is stable, supported or not, I'm not sure., but you're always welcome to raise a support ticket and one of our engineers will kindly continue with the assistance.
