Re: GCP Databricks Spark Connector for Cassandra -...

KaustubhShah · ‎02-25-2025

Hello,

I am using Databricks runtime 12.2 with the spark connector -

com.datastax.spark:spark-cassandra-connector_2.12:3.3.0

as runtime 12.2 comes with spark 3.3.2 and scala 2.12. I encounter an issue with conneciting to cassandra DB using the below code:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("CassandraConnector")\

.config("spark.cassandra.connection.port","9042")\

.config("spark.cassandra.auth.username","user")\

.config("spark.cassandra.auth.password","pass")\

.config("spark.cassandra.connection.ssl.enabled", "false")\

.config("spark.cassandra.connection.host", "hostnma")\

.getOrCreate()

df = spark.read\

.format("org.apache.spark.sql.cassandra")\

.option("keyspace", "name")\

.option("table", "table_name")\

.load()
df.count()

Error: java.io.IOException: Failed to open native connection to Cassandra at {hostname} :: com.typesafe.config.impl.ConfigImpl.newSimpleOrigin(Ljava/lang/String;)Lcom/typesafe/config/ConfigOrigin;

But when i upgrade to runtime 15.4 and use connector: com.datastax.spark:spark-cassandra-connector_2.12:3.5.0

the issue is self resolved.

Can someone please help me diagnosing this issue because I have to keep using 12.2 runtime.

Thanks

cgrant · ‎02-27-2025

Try using the assembly version of the jar with 12.2.

https://mvnrepository.com/artifact/com.datastax.spark/spark-cassandra-connector-assembly

If this doesn't work, please paste the full, original stacktrace

GCP Databricks Spark Connector for Cassandra - Error: com.typesafe.config.impl.ConfigImpl.newSimple