Hi, I haven't got the ability to test this myself but based on some internal research, I think the following is true:

Hi,

The most likely issue is your truststore configuration. Setting spark.driver.extraJavaOptions -Djavax.net.ssl.trustStore=<custom-path> replaces the JVM's entire default truststore rather than extending it. This means the JVM loses all the standard public CAs,
which breaks Ivy/Maven's ability to connect over HTTPS — including to your Artifactory.

Instead of overriding the truststore via Spark conf, add your Artifactory CA to the default Java keystore in your init script:

#!/bin/bash

cat << 'EOF' > /usr/local/share/ca-certificates/artifactory-ca.crt
-----BEGIN CERTIFICATE-----
<your-CA-certificate-chain>
-----END CERTIFICATE-----
EOF

update-ca-certificates

JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:bin/java::")
KEYSTORE="$JAVA_HOME/lib/security/cacerts"

keytool -noprompt -import -trustcacerts \
-alias artifactory-ca \
-keystore $KEYSTORE \
-storepass changeit \
-file /usr/local/share/ca-certificates/artifactory-ca.crt

Then remove the spark.driver.extraJavaOptions and spark.executor.extraJavaOptions truststore settings entirely. The JVM will use the updated default keystore which now has both the standard public CAs and your Artifactory CA.

If this fixes it can you mark as the accepted solution to help others please.

Thanks,

Emma