Issues when configuring keystore spark config for pyspark to mongo atlas X.509 connectivity

[jash281098]

Step followed - 

Step1: To add init script that will copy the keystore file in the tmp location.

Step2: To add spark config in cluster advance options - 

spark.driver.extraJavaOptions -Djavax.net.ssl.keyStore=/tmp/keystore.jks -Djavax.net.ssl.keyStorePassword=<pass>
spark.executor.extraJavaOptions -Djavax.net.ssl.keyStore=/tmp/keystore.jks -Djavax.net.ssl.keyStorePassword=<pass>

After doing above configuration the cluster doesn't execute any commands. In log4j driver logs their is a constant error message. PFA complete error log below. What is the recommended way to achieve mongo atlas X.509 connectivity?

Error message - 

ERROR Datastore: Exception thrown creating StoreManager. See the nested exception
Error creating transactional connection factory

Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "HikariCP" plugin to create a ConnectionPool gave an error : Failed to initialize pool: Could not connect to address=(host=mdb7sywh50xhpr.chkweekm4xjq.us-east-1.rds.amazonaws.com)(port=3306)(type=master) : Could not connect to mdb7sywh50xhpr.chkweekm4xjq.us-east-1.rds.amazonaws.com:3306 : readHandshakeRecord

[mderela]

The error isn’t actually about MongoDB. HikariCP failing on port 3306 via DataNucleus is your Hive Metastore losing its SSL connection to MySQL on the driver.
Setting javax.net.ssl.keyStore globally in extraJavaOptions overwrites the default JVM truststore for every SSL connection on the driver, including the Metastore. The RDS certificate is no longer trusted because your keystore only contains the X.509 key for MongoDB, not the CA for RDS.
One approach worth trying: instead of replacing the truststore, extend it in your init script by importing the MongoDB cert into the existing JVM cacerts rather than pointing to a separate keystore file:

keytool -importcert -keystore $JAVA_HOME/lib/security/cacerts -storepass changeit -alias mongo-atlas -file /tmp/mongo.crt -noprompt

That way the default truststore stays intact for Metastore and everything else, and MongoDB gets its cert alongside it.

as soon as you confirmed that it is working - we can try to deep dive to find the root cause of the issue.