Deepak_Bhutada
Databricks Employee
Databricks Employee

‎12-20-2021 10:21 AM

Hi @Yikun Song​ 

We found an issue with it that will be fixed in the next round of patches (to be released mid-January).

As a workaround,

  1. You need to use the assembly jar like https://repo1.maven.org/maven2/org/mongodb/spark/mongo-spark-connector_2.12/3.0.1/mongo-spark-connec... to also include in transitive dependencies.
  2. After adding the JAR, you need to first run any query, like a “SELECT 1” query to get the JAR properly synced. Otherwise, if spark.read.format(“mongo”) is called directly, a request to use it to resolve the datasource will reach DBR too early, before the library is synced.

So adding the assembly jar to --jars, and first running a SELECT 1 query to make sure that it gets synced to the server should be a temp working workaround.