MongoDB Spark Connector v10.x read error on Databricks 14.3 LTS

[pferreira]

Im facing an error when updating DBR from 13.3 LTS to 14.3LTS

Im using the spark:mongo-spark-connector:10.2.1 and running the following script

connectionString = ******
database = *****
collection = *****


spark = SparkSession \
    .builder \
    .master("local") \
    .appName("myAPP") \
    .config("spark.mongodb.read.connection.uri", connectionString) \
    .config("spark.mongodb.write.connection.uri", connectionString) \
    .config('spark.jars.packages', 'org.mongodb.spark:mongo-spark-connector:10.2.1') \
    .getOrCreate()


caDf = spark.read.format("mongodb") \
    .option("connection.uri", connectionString) \
    .option("database", database) \
    .option("collection", collection)\ 
    .load()

And when I run this on a cluster with the DBR 14.3 LTS I get the following error:

Py4JJavaError: An error occurred while calling o433.showString.
: java.lang.NoClassDefFoundError: Could not initialize class com.mongodb.spark.sql.connector.read.MongoScanBuilder
	at com.mongodb.spark.sql.connector.MongoTable.newScanBuilder(MongoTable.java:121)
	at org.apache.spark.sql.execution.datasources.v2.V2ScanRelationPushDown$$anonfun$createScanBuilder$1.applyOrElse(V2ScanRelationPushDown.scala:59)
	at org.apache.spark.sql.execution.datasources.v2.V2ScanRelationPushDown$$anonfun$createScanBuilder$1.applyOrElse(V2ScanRelationPushDown.scala:57)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:473)
	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:83)

thanks in advance

[feiyun0112]

The following notebook shows you how to read and write data to MongoDB Atlas, 

https://docs.databricks.com/en/_extras/notebooks/source/mongodb.html 

[pferreira]

Hi @feiyun0112 
Thank you for your help.

Unfortunatle, that didn't help...
Our code is like its documented and was working with the DBR 13.3 LTS, however, once we upgraded the cluster to DBR 14.3LTS we got this error.