โ04-11-2023 02:49 PM
Hello,
Iยดve added a library to the cluster and it appears in SPARK UI as Added By User
spark://10.139.64.4:43001/jars/addedFile307892533757162075org_mongodb_spark_mongo_spark_connector_2_12_10_1_1-98946.jarAdded By User
I'm trying to connect using the following SparkSession configuration, but it is not working:
spark = (SparkSession.builder.config('spark.mongodb.input.uri',connectionString).config('spark.jars.packages', 'org.mongodb.spark:mongo-spark-connector_2.12:10.1.1').getOrCreate())
If I uninstall this library and install the previous one the , 2.12:3.0.1, the conection works.
Does anyone can help me with that?
Thanks
โ04-15-2023 06:13 PM
It looks like you are trying to connect to MongoDB using the mongo-spark-connector_2.12:10.1.1 library, but you are facing issues with the connection. Here are a few things you can try to resolve the issue:
$SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client --driver-memory 4g --executor-memory 2g --executor-cores 1 --num-executors 2 --conf spark.eventLog.enabled=true --conf spark.eventLog.dir=hdfs:///spark-history --conf spark.history.fs.logDirectory=hdfs:///spark-history --jars /path/to/mongo-spark-connector_2.12-10.1.1.jar /path/to/your/application.jar
3) Try a different version of the library: If the above two steps don't work, you can try using a different version of the mongo-spark-connector library. You can find the list of available versions here: https://mvnrepository.com/artifact/org.mongodb.spark/mongo-spark-connector_2.12
4) Check compatibility with MongoDB server version: Make sure that the version of mongo-spark-connector library you are using is compatible with the version of MongoDB server you are using. You can check the compatibility matrix here: https://docs.mongodb.com/spark-connector/master/#compatibility-matrix
I hope these suggestions help you resolve the issue.
โ07-26-2023 02:34 PM
In version 10.x of MongoDB Spark Connector some configuration options have changed.
Now you have to pass spark.mongodb.read.connection.uri instead of spark.mongodb.input.uri.
Checkout the new other options in Read Configuration Options โ MongoDB Spark Connector.
โ08-03-2023 03:12 PM
I face similar problem with anything above org.mongodb.spark:mongo-spark-connector_2.12:3.0.1
So version 10+ of org.mongodb.spark:mongo-spark-connector_2.12 from https://mvnrepository.com/artifact/org.mongodb.spark/mongo-spark-connector_2.12 are not working with Databricks 12.2 LTS ๐ฃ
org.apache.spark.SparkClassNotFoundException: [DATA_SOURCE_NOT_FOUND] Failed to find data source: mongo. Please find packages at `https://spark.apache.org/third-party-projects.html`.
Is there anything that should be done additionally to installing it into Library? Maybe some additional cluster option?
โ10-30-2023 04:46 AM
@DmytroSokhach I think it works if you change mongo to mongodb in the options. and use spark.mongodb.read.connection.uri instead of spark.mongodb.input.uri as @silvadev suggested.
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.