Issue with Reading MongoDB Data in Unity Catalog Cluster

naveenprasanth — Fri, 29 Dec 2023 15:55:30 GMT

I am encountering an issue while trying to read data from MongoDB in a Unity Catalog Cluster using PySpark. I have shared my code below:

from pyspark.sql import SparkSession database = "cloud" collection = "data" Scope = "XXXXXXXX" Key = "XXXXXX-YYYYYY-ZZZZZZ" connectionString = dbutils.secrets.get(scope=Scope, key=Key) spark = ( SparkSession.builder.config("spark.mongodb.input.uri", connectionString) .config("spark.mongodb.output.uri", connectionString) .config("spark.jars.packages", "org.mongodb.spark:mongo-spark-connector_2.12:3.2.0") .getOrCreate() ) # Reading from MongoDB df = ( spark.read.format("mongo") .option("uri", connectionString) .option("database", database) .option("collection", collection) .load() )

However, I am encountering the following error:

org.apache.spark.SparkClassNotFoundException: [DATA_SOURCE_NOT_FOUND] Failed to find data source: mongo. Please find packages at `https://spark.apache.org/third-party-projects.html`.

I have already included the necessary MongoDB Spark Connector package, but it seems like Spark is unable to find the data source. Can someone please help me understand what might be causing this issue and how I can resolve it? Any insights or suggestions would be greatly appreciated. Thank you!

Re: Issue with Reading MongoDB Data in Unity Catalog Cluster

Wojciech_BUK — Fri, 29 Dec 2023 23:29:52 GMT

Few points

1. chce if you installed exactly same driver version as you are pointing this in code (2.12:3.2.0) it has to match 100percent

org.mongodb.spark:mongo-spark-connector_2.12:3.2.0

2. I have seen people configuring connction to atlas in two ways

Option 1

Back in Databricks in your cluster configuration, under Advanced Options (bottom of page), paste the connection string for both the spark.mongodb.output.uri and spark.mongodb.input.uri variables. Plase populate the username and password field appropriatly. This way all the workbooks you are running on the cluster will use this configuration.

Option 2:

Alternativley you can explictly set the option when calling APIs like: spark.read.format("mongo").option("spark.mongodb.input.uri", connectionString).load(). If congigured the variables in the cluster, you don't have to set the option.

3. Try with single user cluster ( if you are using shared cluster) .

4. Check mongo db driver compatibility with spark version you are using

Hope that it will help .

topic Re: Issue with Reading MongoDB Data in Unity Catalog Cluster in Data Engineering

Issue with Reading MongoDB Data in Unity Catalog Cluster

Re: Issue with Reading MongoDB Data in Unity Catalog Cluster