Issues when adding keystore spark config for pyspark to mongo atlas X.509 connectivity

[jash281098]

Step followed - 

Step1: To add init script that will copy the keystore file in the tmp location.

Step2: To add spark config in cluster advance options - 

spark.driver.extraJavaOptions -Djavax.net.ssl.keyStore=/tmp/keystore.jks -Djavax.net.ssl.keyStorePassword=<pass>
spark.executor.extraJavaOptions -Djavax.net.ssl.keyStore=/tmp/keystore.jks -Djavax.net.ssl.keyStorePassword=<pass>

After doing above configuration the cluster doesn't execute any commands. In log4j driver logs their is a constant error message. What is the recommended way to achieve mongo atlas X.509 connectivity?

Error Could not connect to mdb7sywh50xhpr.chkweekm4xjq.us-east-1.rds.amazonaws.com:3306 : readHandshakeRecord

[jash281098]

Error Messages -

Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Could not connect to address=(host=mdb7sywh50xhpr.chkweekm4xjq.us-east-1.rds.amazonaws.com)(port=3306)(type=master) : Could not connect to mdb7sywh50xhpr.chkweekm4xjq.us-east-1.rds.amazonaws.com:3306 : readHandshakeRecord

Caused by: java.net.SocketException: Broken pipe (Write failed)

25/05/10 14:19:06 WARN HiveClientImpl: HiveClient got thrift or connection reset exception, destroying client and retrying (13 tries remaining)

Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

Caused by: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory

Caused by: java.lang.reflect.InvocationTargetException

[mark_ott]

To achieve MongoDB Atlas X.509 connectivity from Databricks using PySpark, the standard keystore configuration may fail due to certificate, configuration, or driver method issues. The recommended approach involves several key steps, including properly setting up certificates and Spark configs, and addressing Databricks-specific requirements for secure cluster communication.

Recommended Steps

• Ensure that your keystore and truststore files (JKS or PEM) are valid and contain the correct certificates for X.509 authentication.

• Both keystore and truststore should be imported using JVM tools (keytool for JKS; openssl for PEM).

• Place the files in a secure and accessible cluster location, such as /tmp/, using an init script at cluster startup.​

• Use the following Spark configuration options:

  • spark.driver.extraJavaOptions -Djavax.net.ssl.keyStore=/tmp/keystore.jks -Djavax.net.ssl.keyStorePassword=<password>

  • spark.executor.extraJavaOptions -Djavax.net.ssl.keyStore=/tmp/keystore.jks -Djavax.net.ssl.keyStorePassword=<password>

  • For truststore: add -Djavax.net.ssl.trustStore=/tmp/truststore.jks -Djavax.net.ssl.trustStorePassword=<password>, if required.​

• Confirm that the network access in MongoDB Atlas allows connections from your Databricks cluster IP addresses.​

• Ensure the Atlas database user matches the subject of your client certificate and that the CA certificate is uploaded in Atlas settings.​

Common Pitfalls and Error

• The error readHandshakeRecord is usually caused by misconfigured certificates, mismatched distinguished names, or connecting to the wrong port/service (the error references an RDS hostname/port 3306, which is MySQL—not MongoDB Atlas port). Double-check the cluster endpoint and port.​

• When using PEM files, some libraries require options like tlsCAFile, tlsCertificateKeyFile, etc. For Spark, JKS files and the corresponding JVM options are preferred.​

• Make sure your Databricks runtime version and the MongoDB Spark Connector support X.509 and your Java version.

Best Practices Checklist

• Validate your keystore/truststore with provided Java tooling.

• Mount certificates in all nodes with an init script.

• Use correct Spark configs for both driver and executor.

• Whitelist Databricks cluster IPs in Atlas Network Access.

• Avoid connecting via non-Atlas-hostnames or SQL database ports (such as RDS/MySQL).

If these steps do not resolve the issue, review the Spark and cluster logs for specific SSL/TLS errors, and check MongoDB Atlas user and certificate matching rules in the Atlas dashboard and documentation.​