topic MongoDB Spark Connector v10.x read error on Databricks 14.3 LTS in Data Engineering

MongoDB Spark Connector v10.x read error on Databricks 14.3 LTS

pferreira — Wed, 14 Feb 2024 18:27:45 GMT

Im facing an error when updating DBR from 13.3 LTS to 14.3LTS

Im using the spark:mongo-spark-connector:10.2.1 and running the following script

connectionString = ****** database = ***** collection = ***** spark = SparkSession \ .builder \ .master("local") \ .appName("myAPP") \ .config("spark.mongodb.read.connection.uri", connectionString) \ .config("spark.mongodb.write.connection.uri", connectionString) \ .config('spark.jars.packages', 'org.mongodb.spark:mongo-spark-connector:10.2.1') \ .getOrCreate() caDf = spark.read.format("mongodb") \ .option("connection.uri", connectionString) \ .option("database", database) \ .option("collection", collection)\ .load()

And when I run this on a cluster with the DBR 14.3 LTS I get the following error:

Py4JJavaError: An error occurred while calling o433.showString. : java.lang.NoClassDefFoundError: Could not initialize class com.mongodb.spark.sql.connector.read.MongoScanBuilder at com.mongodb.spark.sql.connector.MongoTable.newScanBuilder(MongoTable.java:121) at org.apache.spark.sql.execution.datasources.v2.V2ScanRelationPushDown$$anonfun$createScanBuilder$1.applyOrElse(V2ScanRelationPushDown.scala:59) at org.apache.spark.sql.execution.datasources.v2.V2ScanRelationPushDown$$anonfun$createScanBuilder$1.applyOrElse(V2ScanRelationPushDown.scala:57) at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:473) at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:83)

thanks in advance

Re: MongoDB Spark Connector v10.x read error on Databricks 14.3 LTS

feiyun0112 — Thu, 15 Feb 2024 02:07:10 GMT

The following notebook shows you how to read and write data to MongoDB Atlas,

https://docs.databricks.com/en/_extras/notebooks/source/mongodb.html

Re: MongoDB Spark Connector v10.x read error on Databricks 14.3 LTS

pferreira — Thu, 15 Feb 2024 19:48:46 GMT

Hi @feiyun0112
Thank you for your help.

Unfortunatle, that didn't help...
Our code is like its documented and was working with the DBR 13.3 LTS, however, once we upgraded the cluster to DBR 14.3LTS we got this error.

Re: MongoDB Spark Connector v10.x read error on Databricks 14.3 LTS

Namrata1 — Wed, 30 Apr 2025 11:03:53 GMT

hey @pferreira,

did you get any solution on your issue?

Re: MongoDB Spark Connector v10.x read error on Databricks 14.3 LTS

pmaferreira — Wed, 30 Apr 2025 17:05:27 GMT

Hi @Namrata1
we end up using other connector

Re: MongoDB Spark Connector v10.x read error on Databricks 14.3 LTS

Namrata1 — Fri, 02 May 2025 15:02:58 GMT

Hi @pmaferreira ,
can you please help with which version you are using and is it supports ignoreNullValues option?