topic Re: Kafka unable to read client.keystore.jks. in Data Engineering

Kafka unable to read client.keystore.jks.

Jayanth746 — Tue, 08 Nov 2022 03:04:31 GMT

Below is the error we have received when trying to read the stream

Caused by: kafkashaded.org.apache.kafka.common.KafkaException: Failed to load SSL keystore /dbfs/FileStore/Certs/client.keystore.jks

Caused by: java.nio.file.NoSuchFileException: /dbfs/FileStore/Certs/client.keyst

When trying to read a stream from Kafka, databricks is unable to find keystore files.

df = spark.readStream \

.format("kafka") \

.option("kafka.bootstrap.servers","kafka server with port") \

.option("kafka.security.protocol", "SSL") \

.option("kafka.ssl.truststore.location",'/dbfs/FileStore/Certs/client.truststore.jks' ) \

.option("kafka.ssl.keystore.location", '/dbfs/FileStore/Certs/client.keystore.jks') \

.option("kafka.ssl.keystore.password", keystore_pass) \

.option("kafka.ssl.truststore.password", truststore_pass) \

.option("kafka.ssl.keystore.type", "JKS") \

.option("kafka.ssl.truststore.type", "JKS") \

.option("subscribe","sports") \

.option("startingOffsets", "earliest") \

.load()

The file exists in the dbfs and also able to read the file.

Re: Kafka unable to read client.keystore.jks.

Jayanth746 — Wed, 09 Nov 2022 19:21:21 GMT

Hi @Debayan Mukherjee , Please see the results after using PEM as the keystore type.

Caused by: kafkashaded.org.apache.kafka.common.errors.InvalidConfigurationException: SSL key store password cannot be specified with PEM format, only key password may be specified

I have use the document posted in the chat to get this working.

Also if I use SASL_SSL as protocol I get the below error

Caused by: java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in the JAAS configuration. System property 'java.security.auth.login.config' is not set

The files are present in the dbfs

Re: Kafka unable to read client.keystore.jks.

Jayanth746 — Tue, 15 Nov 2022 14:09:25 GMT

Do I need to use JAAS even if I already have certificates for SSL connection?

I am only looking to establish SSL connection and not SASL.

Re: Kafka unable to read client.keystore.jks.

Jayanth746 — Wed, 16 Nov 2022 09:46:29 GMT

Hi @Debayan Mukherjee , This worked after using the absolute path

/dbfs/dbfs/FileStore/Certs/client.truststore.jks instead of just dbfs/FileStore/Certs/client.truststore.jks.

However, I need this to be working for a ADLS gen2 path.

Re: Kafka unable to read client.keystore.jks.

Debayan — Wed, 09 Nov 2022 07:11:31 GMT

Hi @Jayanth Goulla , Does this works: kafka.ssl.keystore.type = PEM ?

Reference: https://docs.databricks.com/structured-streaming/kafka.html#use-ssl

Re: Kafka unable to read client.keystore.jks.

Debayan — Fri, 11 Nov 2022 18:01:32 GMT

You’ll have to construct JAAS file and pass with JVM option. Or, you can pass the content of JAAS as Kafka source option, say, dynamic JAAS config.https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+Dynamic+JAAS+configuration+for+Kafka+clients

Re: Kafka unable to read client.keystore.jks.

mwoods — Mon, 18 Sep 2023 16:15:26 GMT

@Jayanth746did you have any luck with this eventually? Hitting the same issue - appears that spark isn't able to read from adls directly, but the docs are vague as to whether it should be possible. Looks like will probably have to copy them to a local path first.

Re: Kafka unable to read client.keystore.jks.

Jayanth746 — Mon, 18 Sep 2023 16:32:11 GMT

Hi @mwoods , I was unable to refer to ADLS path directly.

This is what I have done to get this working

val keystore_location = adls_path + "/" + operator + "/certs/client.keystore.jks"

val dbfs_ks_location = "dbfs:/FileStore/"+ operator +"/Certs/client.keystore.jks"

dbutils.fs.cp(keystore_location,dbfs_ks_location)

.option("kafka.ssl.keystore.location","/"+dbfs_ks_location.replace(":",""))

Re: Kafka unable to read client.keystore.jks.

mwoods — Fri, 22 Sep 2023 14:24:35 GMT

@Jayanth746- FYI, as of today, reading the keystore/truststore from abfss paths directly is now working for me, so may be worth a retry on your end.

Not sure whether it was fixed on the DataBricks side, or if it was down to a change of setup on my side. If you find it still doesn't work for you, assuming you have used an external location to access, double-check that the principal/grant mapping in there is correct.

Re: Kafka unable to read client.keystore.jks.

mwoods — Fri, 22 Sep 2023 19:13:13 GMT

Ok, scrub that - the problem in my case was that I was using the 14.0 databricks runtime, which appears to have a bug relating to abfss paths here. Switching back to the 13.3 LTS release resolved it for me. So if you're in the same boat finding abfss paths in kafka.ssl.keystore.location and kafka.ssl.truststore.location are failing, try switching back to 13.3 LTS.