topic Re: "No SRV records" intermittent error when running Databricks Pyspark to connect Mongo Atlas in Data Engineering

"No SRV records" intermittent error when running Databricks Pyspark to connect Mongo Atlas

200723 — Wed, 26 Oct 2022 20:08:05 GMT

My Mongo Atlas connect url is like mongodb+srv://<srv_hostname>

I don't want to use direct url like mongodb://<hostname1, hostname2, hostname3....> because our Mongo Atlas global clusters have many hosts. It would be hard to maintain.

Our java programs in our GCP cloud connecting to the same Mongo Atlas SRV do not have this issue. I suspect something wrong with our Databricks or pyspark configs or driver versions compatibility. Please help.

level: ERROR

message: Exception occured: %s Traceback (most recent call last):

File "<command-389574246471033>", line 9, in <module>

if df.count() != 0:

File "/databricks/spark/python/pyspark/sql/dataframe.py", line 688, in count

return int(self._jdf.count())

File "/databricks/spark/python/lib/py4j-0.10.9.1-src.zip/py4j/java_gateway.py", line 1304, in __call__

return_value = get_return_value(

File "/databricks/spark/python/pyspark/sql/utils.py", line 117, in deco

return f(*a, **kw)

File "/databricks/spark/python/lib/py4j-0.10.9.1-src.zip/py4j/protocol.py", line 326, in get_return_value

raise Py4JJavaError(

py4j.protocol.Py4JJavaError: An error occurred while calling o468.count.

: com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches com.mongodb.client.internal.MongoClientDelegate$1@43eaa290. Client view of cluster state is {type=REPLICA_SET, srvResolutionException=com.mongodb.MongoConfigurationException: No SRV records available for _mongodb._tcp.mongo-core-coreapp-clus-pri.tpl3f.mongodb.net, servers=[]

at com.mongodb.internal.connection.BaseCluster.createTimeoutException(BaseCluster.java:424)

Re: "No SRV records" intermittent error when running Databricks Pyspark to connect Mongo Atlas

200723 — Fri, 28 Oct 2022 13:42:20 GMT

Our code is already as same as the one which is mentioned in the article link which you have shared.

Re: "No SRV records" intermittent error when running Databricks Pyspark to connect Mongo Atlas

Anonymous — Mon, 28 Nov 2022 06:47:10 GMT

Hi @Raymond Lai

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: "No SRV records" intermittent error when running Databricks Pyspark to connect Mongo Atlas

Noopur_Nigam — Wed, 30 Nov 2022 15:39:14 GMT

Hi @Raymond Lai The issue looks to be on the Mongo DB connector. The connection is created and maintained by the mongo-spark connector. You can try using the direct mongodb hosts in the connection string instead of SRV to avoid doing DNS lookups or you can reach out to MongoDb team for the resolution.