MongoDB Spark Connector v10.x read error on Databricks 14.3 LTS

pferreira
Databricks Partner

Im facing an error when updating DBR from 13.3 LTS to 14.3LTS

Im using the spark:mongo-spark-connector:10.2.1 and running the following script

 

 

 

connectionString = ******
database = *****
collection = *****


spark = SparkSession \
    .builder \
    .master("local") \
    .appName("myAPP") \
    .config("spark.mongodb.read.connection.uri", connectionString) \
    .config("spark.mongodb.write.connection.uri", connectionString) \
    .config('spark.jars.packages', 'org.mongodb.spark:mongo-spark-connector:10.2.1') \
    .getOrCreate()


caDf = spark.read.format("mongodb") \
    .option("connection.uri", connectionString) \
    .option("database", database) \
    .option("collection", collection)\ 
    .load()

 

 

 

And when I run this on a cluster with the DBR 14.3 LTS I get the following error:

 

 

 

Py4JJavaError: An error occurred while calling o433.showString.
: java.lang.NoClassDefFoundError: Could not initialize class com.mongodb.spark.sql.connector.read.MongoScanBuilder
	at com.mongodb.spark.sql.connector.MongoTable.newScanBuilder(MongoTable.java:121)
	at org.apache.spark.sql.execution.datasources.v2.V2ScanRelationPushDown$$anonfun$createScanBuilder$1.applyOrElse(V2ScanRelationPushDown.scala:59)
	at org.apache.spark.sql.execution.datasources.v2.V2ScanRelationPushDown$$anonfun$createScanBuilder$1.applyOrElse(V2ScanRelationPushDown.scala:57)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:473)
	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:83)

 

 

 

thanks in advance