cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

"No SRV records" intermittent error when running Databricks Pyspark to connect Mongo Atlas

200723
New Contributor II

My Mongo Atlas connect url is like mongodb+srv://<srv_hostname>

I don't want to use direct url like mongodb://<hostname1, hostname2, hostname3....> because our Mongo Atlas global clusters have many hosts. It would be hard to maintain.

Our java programs in our GCP cloud connecting to the same Mongo Atlas SRV do not have this issue. I suspect something wrong with our Databricks or pyspark configs or driver versions compatibility. Please help.

   level: ERROR

   message: Exception occured: %s Traceback (most recent call last):

File "<command-389574246471033>", line 9, in <module>

if df.count() != 0:

File "/databricks/spark/python/pyspark/sql/dataframe.py", line 688, in count

return int(self._jdf.count())

File "/databricks/spark/python/lib/py4j-0.10.9.1-src.zip/py4j/java_gateway.py", line 1304, in __call__

return_value = get_return_value(

File "/databricks/spark/python/pyspark/sql/utils.py", line 117, in deco

return f(*a, **kw)

File "/databricks/spark/python/lib/py4j-0.10.9.1-src.zip/py4j/protocol.py", line 326, in get_return_value

raise Py4JJavaError(

py4j.protocol.Py4JJavaError: An error occurred while calling o468.count.

: com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches com.mongodb.client.internal.MongoClientDelegate$1@43eaa290. Client view of cluster state is {type=REPLICA_SET, srvResolutionException=com.mongodb.MongoConfigurationException: No SRV records available for _mongodb._tcp.mongo-core-coreapp-clus-pri.tpl3f.mongodb.net, servers=[]

at com.mongodb.internal.connection.BaseCluster.createTimeoutException(BaseCluster.java:424)

3 REPLIES 3

200723
New Contributor II

Our code is already as same as the one which is mentioned in the article link which you have shared.

Anonymous
Not applicable

Hi @Raymond Lai​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

Noopur_Nigam
Valued Contributor II

Hi @Raymond Lai​  The issue looks to be on the Mongo DB connector. The connection is created and maintained by the mongo-spark connector. You can try using the direct mongodb hosts in the connection string instead of SRV to avoid doing DNS lookups or you can reach out to MongoDb team for the resolution.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group