Spark Streaming with SASL_SSL Kafka throwing java.nio.file.NoSuchFileException: dbfs:/mnt/**/kafka.client.truststore.imported.jks

mj2022 — Thu, 30 Jun 2022 12:51:43 GMT

I testing Spark Streaming working withSASL_SSL enabled kafka broker in a notebook.

as per this guide https://docs.databricks.com/spark/latest/structured-streaming/kafka.html

i have copied jsk files in an s3 bucket and mounted it in dbfs.

In notebook when i run dbutils.fs.head( jksPath), i can see file contents.

But when i run the spark streaming job throwing.

Caused by: java.nio.file.NoSuchFileException: dbfs:/mnt/xxxx/kafka.client.truststore.imported.jks
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
	at java.nio.file.Files.newByteChannel(Files.java:361)
	at java.nio.file.Files.newByteChannel(Files.java:407)
	at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
	at java.nio.file.Files.newInputStream(Files.java:152)
	at kafkashaded.org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:370)kafkashaded.org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:138)
	at kafkashaded.org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:95)
	at kafkashaded.org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:180)

I also installed org.apache.spark:spark-sql-kafka-0-10_2.12:3.2.1 in databricks cluster.

Re: Spark Streaming with SASL_SSL Kafka throwing java.nio.file.NoSuchFileException: dbfs:/mnt/**/kafka.client.truststore.imported.jks

Prabakar — Thu, 30 Jun 2022 14:28:46 GMT

Looking into it.

Re: Spark Streaming with SASL_SSL Kafka throwing java.nio.file.NoSuchFileException: dbfs:/mnt/**/kafka.client.truststore.imported.jks

mj2022 — Tue, 26 Jul 2022 12:48:10 GMT

Thanks..Yes '/dbfs/mnt/xxxx/kafka.client.truststore.imported.jks' path worked.

Also other workaround we got it working, is copy the file from s3 to filesystem using init script and use filepath.

topic Re: Spark Streaming with SASL_SSL Kafka throwing java.nio.file.NoSuchFileException: dbfs:/mnt/**/kafka.client.truststore.imported.jks in Data Engineering

Spark Streaming with SASL_SSL Kafka throwing java.nio.file.NoSuchFileException: dbfs:/mnt/**/kafka.client.truststore.imported.jks

Re: Spark Streaming with SASL_SSL Kafka throwing java.nio.file.NoSuchFileException: dbfs:/mnt/**/kafka.client.truststore.imported.jks

Re: Spark Streaming with SASL_SSL Kafka throwing java.nio.file.NoSuchFileException: dbfs:/mnt/**/kafka.client.truststore.imported.jks