Hello,
I am using Databricks runtime 12.2 with the spark connector -
com.datastax.spark:spark-cassandra-connector_2.12:3.3.0
as runtime 12.2 comes with spark 3.3.2 and scala 2.12. I encounter an issue with conneciting to cassandra DB using the below code:
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("CassandraConnector")\
.config("spark.cassandra.connection.port","9042")\
.config("spark.cassandra.auth.username","user")\
.config("spark.cassandra.auth.password","pass")\
.config("spark.cassandra.connection.ssl.enabled", "false")\
.config("spark.cassandra.connection.host", "hostnma")\
.getOrCreate()
df = spark.read\
.format("org.apache.spark.sql.cassandra")\
.option("keyspace", "name")\
.option("table", "table_name")\
.load()
df.count()
Error:
java.io.IOException: Failed to open native connection to Cassandra at {hostname} :: com.typesafe.config.impl.ConfigImpl.newSimpleOrigin(Ljava/lang/String;)Lcom/typesafe/config/ConfigOrigin;
But when i upgrade to runtime 15.4 and use connector: com.datastax.spark:spark-cassandra-connector_2.12:3.5.0the issue is self resolved.
Can someone please help me diagnosing this issue because I have to keep using 12.2 runtime.
Thanks
