Spark Streaming with SASL_SSL Kafka throwing java.nio.file.NoSuchFileException: dbfs:/mnt/**/kafka.client.truststore.imported.jks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-30-2022 05:51 AM
I testing Spark Streaming working withSASL_SSL enabled kafka broker in a notebook.
as per this guide https://docs.databricks.com/spark/latest/structured-streaming/kafka.html
i have copied jsk files in an s3 bucket and mounted it in dbfs.
In notebook when i run dbutils.fs.head( jksPath), i can see file contents.
But when i run the spark streaming job throwing.
Caused by: java.nio.file.NoSuchFileException: dbfs:/mnt/xxxx/kafka.client.truststore.imported.jks
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.newByteChannel(Files.java:407)
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
at java.nio.file.Files.newInputStream(Files.java:152)
at kafkashaded.org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:370)kafkashaded.org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:138)
at kafkashaded.org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:95)
at kafkashaded.org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:180)I also installed org.apache.spark:spark-sql-kafka-0-10_2.12:3.2.1 in databricks cluster.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-30-2022 07:28 AM
Looking into it.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-26-2022 05:48 AM
Thanks..Yes '/dbfs/mnt/xxxx/kafka.client.truststore.imported.jks' path worked.
Also other workaround we got it working, is copy the file from s3 to filesystem using init script and use filepath.