Failed to create new KafkaAdminClient

[FilipezAR]

I want to create connections to kafka with spark.readStream using the following parameters:

kafkaParams = {
    "kafka.sasl.jaas.config": f'org.apache.kafka.common.security.plain.PlainLoginModule required username="{kafkaUsername}" password="{kafkaPassword}";',
    "kafka.sasl.mechanism": "PLAIN",
    "kafka.security.protocol": "SASL_PLAINTEXT",
    "kafka.bootstrap.servers": kafkaBootstrapServers,
    "subscribe": kafkaTopic,
    "group.id": kafkaGroupID
}

I get the following error: kafkashaded.org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient

The problem is that it doesn't see: org.apache.kafka.common.security.plain.PlainLoginModule

But, if I use the library for python (confluent_kafka) everything works correctly with the same parameters inside Notebook Databricks.

Maybe someone has an idea about this?

[Kaniz_Fatma]

Hi @FilipezAR,

• The error message suggests that the class org.apache.kafka.common.security.plain.PlainLoginModule is not found. This class is part of the Kafka clients JAR, and it should be included in the classpath of your Spar...applications.
• There might be compatibility issues between the versions of Kafka and Spark you’re using¹.
• Make sure that the Kafka server is configured correctly to use SASL_PLAINTEXT. You might need to set listener.name.sasl_plaintext.plain.sasl.jaas.config instead of sasl.jaas.confi...².
• Sometimes the issue might also be with Firewall or DNS in BootStrap servers. The Consumer should be able to Reach the Kafka Broker Host. Try pinging the Host to check if any Firewall Blockage³.

If none of these suggestions work, you might want to consider using the Confluent Kafka Python library, which you mentioned is working correctly. 

Remember to replace all placeholders like {kafkaUsername}, {kafkaPassword}, kafkaBootstrapServers, kafkaTopic, and kafkaGroupID with actual values. If you’re already doing this, please ignore this reminder.

Please let me know if you need further assistance! 😊