"No SRV records" intermittent error when running Databricks Pyspark to connect Mongo Atlas

[200723]

My Mongo Atlas connect url is like mongodb+srv://<srv_hostname>

I don't want to use direct url like mongodb://<hostname1, hostname2, hostname3....> because our Mongo Atlas global clusters have many hosts. It would be hard to maintain.

Our java programs in our GCP cloud connecting to the same Mongo Atlas SRV do not have this issue. I suspect something wrong with our Databricks or pyspark configs or driver versions compatibility. Please help.

   level: ERROR

   message: Exception occured: %s Traceback (most recent call last):

File "<command-389574246471033>", line 9, in <module>

if df.count() != 0:

File "/databricks/spark/python/pyspark/sql/dataframe.py", line 688, in count

return int(self._jdf.count())

File "/databricks/spark/python/lib/py4j-0.10.9.1-src.zip/py4j/java_gateway.py", line 1304, in __call__

return_value = get_return_value(

File "/databricks/spark/python/pyspark/sql/utils.py", line 117, in deco

return f(*a, **kw)

File "/databricks/spark/python/lib/py4j-0.10.9.1-src.zip/py4j/protocol.py", line 326, in get_return_value

raise Py4JJavaError(

py4j.protocol.Py4JJavaError: An error occurred while calling o468.count.

: com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches com.mongodb.client.internal.MongoClientDelegate$1@43eaa290. Client view of cluster state is {type=REPLICA_SET, srvResolutionException=com.mongodb.MongoConfigurationException: No SRV records available for _mongodb._tcp.mongo-core-coreapp-clus-pri.tpl3f.mongodb.net, servers=[]

at com.mongodb.internal.connection.BaseCluster.createTimeoutException(BaseCluster.java:424)

[Kaniz]

Hi @Raymond Lai​, Please check this S.O thread which guides you through a step-by-step process on how to connect to MongoDB Atlas from the Databricks cluster using pyspark. Also, please check the code and let us know if that helps.

[200723]

Our code is already as same as the one which is mentioned in the article link which you have shared.

[Anonymous]

Hi @Raymond Lai​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

[Noopur_Nigam]

Hi @Raymond Lai​  The issue looks to be on the Mongo DB connector. The connection is created and maintained by the mongo-spark connector. You can try using the direct mongodb hosts in the connection string instead of SRV to avoid doing DNS lookups or you can reach out to MongoDb team for the resolution.