Databricks Community

pferreira · ‎02-14-2024

Im facing an error when updating DBR from 13.3 LTS to 14.3LTS

Im using the spark:mongo-spark-connector:10.2.1 and running the following script

connectionString = ******
database = *****
collection = *****


spark = SparkSession \
    .builder \
    .master("local") \
    .appName("myAPP") \
    .config("spark.mongodb.read.connection.uri", connectionString) \
    .config("spark.mongodb.write.connection.uri", connectionString) \
    .config('spark.jars.packages', 'org.mongodb.spark:mongo-spark-connector:10.2.1') \
    .getOrCreate()


caDf = spark.read.format("mongodb") \
    .option("connection.uri", connectionString) \
    .option("database", database) \
    .option("collection", collection)\ 
    .load()

And when I run this on a cluster with the DBR 14.3 LTS I get the following error:

Py4JJavaError: An error occurred while calling o433.showString.
: java.lang.NoClassDefFoundError: Could not initialize class com.mongodb.spark.sql.connector.read.MongoScanBuilder
	at com.mongodb.spark.sql.connector.MongoTable.newScanBuilder(MongoTable.java:121)
	at org.apache.spark.sql.execution.datasources.v2.V2ScanRelationPushDown$$anonfun$createScanBuilder$1.applyOrElse(V2ScanRelationPushDown.scala:59)
	at org.apache.spark.sql.execution.datasources.v2.V2ScanRelationPushDown$$anonfun$createScanBuilder$1.applyOrElse(V2ScanRelationPushDown.scala:57)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:473)
	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:83)

thanks in advance

feiyun0112 · ‎02-14-2024

The following notebook shows you how to read and write data to MongoDB Atlas,

https://docs.databricks.com/en/_extras/notebooks/source/mongodb.html

pferreira · ‎02-15-2024

Hi @feiyun0112
Thank you for your help.

Unfortunatle, that didn't help...
Our code is like its documented and was working with the DBR 13.3 LTS, however, once we upgraded the cluster to DBR 14.3LTS we got this error.